Lead/Senior Data Engineer

eClerx LLC; Pune District, Mumbai; Hybrid; INR 2,500,000 - 4,000,000; Full time

Job summary

eClerx LLC seeks a Senior Data Engineer to design, build, and maintain scalable data platforms and pipelines across batch and streaming workloads. You will work with data lakes/warehouses, orchestration, and data quality while partnering with business and tech stakeholders.

The role requires hands-on SQL/PostgreSQL and Python, experience with Airflow or Dagster, and strong AWS data services knowledge, including S3 and Glue. Collaboration and governance are key to success.

Qualifications

• 6–8 years of data engineering experience required, with distributed data systems background.
• Strong hands-on SQL and Python for data processing and pipeline development.
• Experience with Airflow or Dagster for orchestration and data workflows.
• Solid knowledge of AWS data services including S3 and Glue.
• Familiarity with streaming tech: Kafka, Kinesis, Spark Streaming.
• Data modeling, schema design, and scalable processing architectures.
• Understanding of data quality, governance, and pipeline monitoring.
• Excellent problem-solving and communication skills.

Responsibilities

• Design, build, and maintain scalable ETL/ELT pipelines for batch and streaming data.
• Develop data lake and data warehouse architectures optimized for performance and cost.
• Orchestrate data workflows using tools like Airflow or Dagster.
• Ensure data quality, integrity, governance, and reliability across pipelines.
• Optimize SQL queries, storage layouts, and processing jobs for scale.
• Collaborate with stakeholders to translate data requirements into solutions.
• Drive best practices in data modeling, schema design, testing, and version control.
• Troubleshoot and improve pipelines to ensure high availability.

Skills

SQL/PostgreSQL

Python

Airflow

Dagster

AWS

Kafka

Kinesis

Spark Structured Streaming

Docker

Kubernetes

CI/CD

Infrastructure as Code

dbt

Tools

Airflow

Dagster

Amazon S3

AWS Glue

Apache Kafka

Amazon Kinesis

Spark Structured Streaming

Docker

Kubernetes

dbt

CI/CD

Terraform

Job description

Lead/Senior Data Engineer

Experience: 6 to 8 years
Employment Type: Full-time

About the Role

We are looking for an experienced Senior Data Engineer to design, build, and maintain scalable data platforms and pipelines. You will work across batch and streaming workloads, data lakes/warehouses, orchestration, and data quality while partnering closely with business and technical stakeholders.

Key Responsibilities

• Design, develop, and maintain robust, scalable ETL/ELT pipelines for batch and streaming data.
• Build and manage data lake and data warehouse architectures optimized for performance, reliability, and cost.
• Develop and orchestrate data workflows using modern orchestration tools such as Airflow or Dagster.
• Ensure data quality, integrity, governance, and reliability across pipelines and platforms.
• Optimize SQL queries, storage layouts, and data processing jobs for performance at scale.
• Collaborate with business and technical stakeholders to understand data requirements and translate them into scalable solutions.
• Drive best practices around data modeling, schema design, testing, documentation, and version control.
• Troubleshoot and improve data pipelines to ensure high availability and operational reliability.

Required Skills & Experience

• 610 years of experience in data engineering, with a strong foundation in distributed data systems.
• Strong hands-on expertise in SQL/PostgreSQL and Python for data processing and pipeline development.
• Experience with workflow orchestration tools such as Apache Airflow, Dagster, or similar.
• Strong working knowledge of AWS data services, particularly Amazon S3 and AWS Glue.
• Familiarity with streaming technologies such as Apache Kafka, Amazon Kinesis, or Spark Structured Streaming.
• Experience with data modeling, schema design, and scalable data processing architectures.
• Strong understanding of data quality, reliability, and pipeline monitoring.
• Excellent problem-solving and communication skills.

Good to Have

• Experience with dbt and modern transformation workflows.
• Experience with data quality and observability tools.
• Knowledge of Docker and Kubernetes.
• Understanding of data security, compliance, and governance frameworks.
• Experience working with CI/CD and infrastructure-as-code practices.

What You'll Bring

• A strong engineering mindset with a focus on building reliable, scalable systems.
• Ability to work independently while collaborating effectively with cross-functional teams.
• Strong ownership of data platforms and a commitment to engineering best practices.
• Curiosity and willingness to learn and adopt modern data technologies.