Lead Data Engineer

Lead Data Engineer

  • Long term contract
  • India

Pyspark, Snowflake, Scala, Hives, Kafka, Python, AWS

  • Build High level technical design both for Streaming and batch processing systems
  • Design and build reusable components, frameworks and libraries at scale to support analytics data products
  • Perform POCs on new technology, architecture patterns
  • Design and implement product features in collaboration with business and Technology stakeholders
  • Anticipate, identify and solve issues concerning data management to improve data quality
  • Clean, prepare and optimize data at scale for ingestion and consumption
  • Drive the implementation of new data management projects and re-structure of the current data architecture
  • Implement complex automated workflows and routines using workflow scheduling tools
  • Build continuous integration, test-driven development and production deployment frameworks
  • Drive collaborative reviews of design, code, test plans and dataset implementation performed by other data engineers in support of maintaining data engineering standards
  • Analyze and profile data for the purpose of designing scalable solutions
  • Troubleshoot complex data issues and perform root cause analysis to proactively resolve product and operational issues
  • Lead, Mentor and develop other Sr Data Engineers and Data engineers in adopting best practices and deliver data products.
  • Partner closely with product management to understand business requirements, breakdown Epics,
  • Partner with Engineering Managers to define technology roadmaps, align on design, architecture and enterprise strategy
  • Expert level expertise in building big data solutions
  • Hands-on experience building cloud scalable, real time and high-performance data lake solutions using AWS, EMR, S3, Hive & Spark, Athena
  • Hands-on experience in delivering batch and streaming jobs
  • Expertise in an agile and iterative model
  • Expert level expertise relational SQL
  • Experience with scripting languages such as Shell, Python
  • Experience with source control tools such as GitHub and related dev process
  • Experience with workflow sc
  • Scheduling tools like Airflow
  • In-depth understanding of micro services architecture
  • Strong understanding of developing complex data solutions
  • Experience working on end-to-end solution design
  • Able to lead others in solving complex Data and Analytics problems
  • Strong understanding of data structures and algorithms
  • Strong hands-on experience in solution and technical design
  • Has a strong problem solving and analytical mindset
  • Able to influence and communicate effectively, both verbally and written, with team members and business stakeholders
  • Able to quickly pick up new programming languages, technologies, and frameworks

Resumes to be sent  torecruitment@kloud9.nyc

Apply Online

Position : Lead Data Engineer