Job Summary:
We are seeking an experienced and motivated Data Engineering Lead to join our growing team. The ideal candidate will have a minimum of 3 years of hands‑on experience in data engineering, with a strong focus on Spark. As a Data Engineering Lead, you will play a pivotal role in designing, developing, and maintaining our data infrastructure, as well as leading a team of data engineers to deliver high‑quality solutions.
Roles & Responsibilities:
- Lead a team of data engineers in designing, building, and maintaining robust and scalable data pipelines and ELT/ ETL processes using Spark.
- Collaborate closely with cross‑functional teams including data scientists, analysts, and software engineers to understand data requirements and deliver effective solutions.
- Architect and optimize data storage and processing solutions for performance, reliability, and scalability.
- Design and implement data quality and data validation processes to ensure the accuracy and integrity of our data.
- Stay up-to-date with the latest industry trends and best practices in data engineering and big data technologies.
- Mentor and guide team members, providing technical leadership, code reviews, and knowledge sharing.
- Participate in the full development lifecycle, from requirements gathering to deployment and monitoring.
- Troubleshoot and resolve data pipeline issues, performance bottlenecks, and other technical challenges.
- Develop and maintain documentation for data pipelines, processes, and procedures.
Skill Required:
- Strong expertise in Apache Spark for processing and analyzing large datasets.
- Experience of working with EMR and DataProc is a plus.
- Strong coding experinence in programming languages- Python, or Java.
- Solid understanding of distributed computing concepts and data partitioning techniques.
- Experience with cloud platforms such as AWS, GCP, or Azure and their data services.
- Proven track record of designing and implementing ELT/ ETL processes and data pipelines.
- Experience with data modeling, data warehousing using BigQuery, and data architecture principles.
- Experience or Familiarity with LakeHouse architecture with Apache delta files and iceberg.
- Experience with Apache Kafka, Apache Airflow, Debezium and Kafka Connect on scale.
- Experience with zero code or low code data pipelines using open source.
- Experience of deploying data pipelines using Containers and Kubernetes.
- Excellent problem‑solving skills and the ability to work in a fast‑paced, collaborative environment.
- Strong leadership and communication skills, with the ability to mentor and guide team members.
- Experience with version control systems (e.g., Git) and agile development methodologies.
- Knowledge of data security and privacy best practices is a plus.
Education: B.E/B.Tech (CS/IT) preferred
Location: Noida 135 (WFO)