Lead Data Engineer

Recro; Pune District; Sur place; INR 3 000 000 - 6 000 000; Plein temps

Résumé du poste

Recro in Pune is seeking a Lead Data Engineer to design and build scalable batch and streaming data pipelines, leverage Lakehouse/Data Warehouse platforms, and collaborate with analytics teams to empower data-driven decisions.

You will lead data modeling, optimize Spark jobs, ensure data quality, governance, and reliability, and mentor engineers, while applying CI/CD practices and working onsite in Pune.

Qualifications

• Bachelor's degree in Computer Science, Engineering, or a related field.
• 8+ years of hands-on experience in Data Engineering.
• Strong expertise in Python, Scala, or Java and high proficiency in SQL.
• Deep experience with Apache Spark (PySpark/Scala) and Databricks.
• Familiarity with cloud platforms (AWS, Azure, or GCP) and data warehouses (Snowflake, Redshift, etc).
• Sound knowledge of data modeling principles, ETL design patterns, and big data frameworks.

Responsabilités

• Data Pipeline Development: Design, build, and maintain high-performance batch and streaming pipelines using PySpark/Scala and Databricks.
• Data Modeling & Architecture: Design efficient schemas and curated datasets optimized for speed, scalability, and cost across Lakehouse/Data Warehouse platforms.
• Data Integration: Seamlessly integrate data from APIs, databases, flat files, and streaming sources across cloud platforms.
• Performance Optimization: Identify bottlenecks, tune Spark configurations, and optimize queries for large-scale data processing.
• Data Quality & Governance: Establish data validation checks and quality standards to maintain reliable data platforms.
• Collaboration & Leadership: Partner with Data Scientists, Analysts, and Cloud Engineers while setting technical best practices, version control, and CI/CD standards.

Connaissances

Python

SQL

Databricks

Spark/PySpark

Scala

Java

ETL

Formation

Bachelor's degree in Computer Science, Engineering, or related field

Outils

Databricks

Apache Spark

Snowflake

Redshift

Description du poste

We are Hiring! Lead Data Engineer | Pune - Onsite

Experience: 8+ Years

Skills: Python, SQL, Databricks, Spark/Pyspark, ETL

Role Overview:

As a Lead Data Engineer, you will drive the design and development of scalable batch & streaming data pipelines, work with modern Lakehouse/Data Warehouse platforms, and partner with analytics and business teams to enable data-driven decision-making.

Key Responsibilities:

• Data Pipeline Development: Design, build, and maintain high-performance batch and streaming pipelines using PySpark/Scala and Databricks.
• Data Modeling & Architecture: Design efficient schemas and curated datasets optimized for speed, scalability, and cost across Lakehouse/Data Warehouse platforms.
• Data Integration: Seamlessly integrate data from APIs, databases, flat files, and streaming sources across cloud platforms.
• Performance Optimization: Identify bottlenecks, tune Spark configurations, and optimize queries for large-scale data processing.
• Data Quality & Governance: Establish data validation checks and quality standards to maintain reliable data platforms.
• Collaboration & Leadership: Partner with Data Scientists, Analysts, and Cloud Engineers while setting technical best practices, version control, and CI/CD standards.

Key Qualifications & Requirements:

• Education: Bachelor's degree in Computer Science, Engineering, or a related field.
• Experience: 8+ years of hands-on experience in Data Engineering.
• Technical Skills:
• Strong expertise in Python, Scala, or Java and high proficiency in SQL.
• Deep experience with Apache Spark (PySpark/Scala) and Databricks.
• Familiarity with cloud platforms (AWS, Azure, or GCP) and data warehouses (Snowflake, Redshift, etc.).
• Sound knowledge of data modeling principles, ETL design patterns, and big data frameworks.

Note : A minimum of 8 years of Data Engineering experience is required for this role.