Lead Data Engineer / Data Engineer (Python)

RapidBrains; Pune District; On-site; INR 1,800,000 - 2,800,000; Part time

Job summary

RapidBrains seeks an experienced Lead Data Engineer / Data Engineer to join our team in Pune. You will design and implement scalable data pipelines, lead architectural decisions and drive best practices across data platforms. Strong Python, Scala and Spark skills are required for handling large-scale data processing.

The role emphasizes cloud-based batch and streaming pipelines, data quality, governance and collaboration with cross-functional teams. Immediate joiners encouraged to apply.

Qualifications

• 8+ years of data engineering experience.
• Strong knowledge of Python, Scala and Spark.

Responsibilities

• Lead the design, development, deployment and maintenance of scalable data pipelines.
• Define data engineering architecture, technical strategies and best practices.
• Build optimized batch and streaming data pipelines in cloud environments.
• Design and maintain ETL, Data Lake and database pipeline architectures.
• Develop solutions for data validation, analysis, monitoring and alerting.
• Work with large datasets and write highly optimized SQL queries.
• Ensure data quality, reliability, governance, security and lifecycle management.
• Review code and drive performance, scalability and engineering best practices.
• Collaborate with Product, Business, QA and Engineering teams.
• Mentor engineers through technical discussions, code reviews and knowledge sharing.
• Troubleshoot complex technical issues and drive continuous improvement.

Skills

Python

Scala

Spark

SQL

Job description

Experience: 8+ Years

Contract: 6 Months

Notice Period : Immediate Joiners

We are looking for an experienced Lead Data Engineer / Data Engineer with strong expertise in Python, Scala, Spark, and large-scale data engineering to join our team.

Key Responsibilities

• Lead the design, development, deployment, and maintenance of scalable data pipelines.
• Define data engineering architecture, technical strategies, and best practices.
• Build optimized batch and streaming data pipelines in cloud environments.
• Design and maintain ETL, Data Lake, and database pipeline architectures.
• Develop solutions for data validation, analysis, monitoring, and alerting.
• Work with large datasets and write highly optimized SQL queries.
• Ensure data quality, reliability, governance, security, and lifecycle management.
• Review code and drive performance, scalability, and engineering best practices.
• Collaborate with Product, Business, QA, and Engineering teams.
• Mentor engineers through technical discussions, code reviews, and knowledge sharing.
• Troubleshoot complex technical issues and drive continuous improvement.