Senior Data Engineer
Senior Data Engineer
Pyspark, Snowflake, Scala, Hives, Kafka, Python, AWS
- Leveraging your deep knowledge to provide technical leadership to take projects from zero to completion.
- Architect, build and maintain scalable data pipelines and access patterns related to permissions and security.
- Research, evaluate and utilize new technologies/tools/frameworks centered around high-volume data processing
- Drive a culture of collaborative reviews of designs, code and test plans
- Work with the architecture engineering team to ensure quality solutions are implemented and engineering best practices adhered to
- Develop set process for Data mining, Data modeling, and Data production
- A data engineer with strong programming experience in Python
- Extensive experience related to processing frameworks such as Spark, Spark Streaming, Airflow, Hive, Sqoop, Kafka etc.
- Experience with big data processing within cloud environments such AWS S3.
- Passion for Data Privacy and Security, Data Management.
- Experience building and shipping data production pipelines sourcing data from a diverse array of sources.
- Deep understanding of measuring and ensuring data quality at scale and the required tooling to monitor and optimize the performance of our data pipelines
- Ability to influence and communicate effectively with team members and business stakeholders
Nice to have: - Knowledge of the retail and eCommerce sector and its use cases.
- Scala
- Oracle/MySQL
- Airflow
- Spark
- AWS Cloud (EMR, S3, SNS,SQS)
- 3 Data Lake
- Kafka
