Senior Data Engineer

DTDC Express; Gurugram District; On-site; INR 2,500,000 - 5,500,000; Full time

Job summary

DTDC Express in Gurgaon is seeking a Senior Data Engineer to design and deploy scalable data pipelines, both batch and streaming, using Python, Kafka, Spark, and AWS. You will build cloud-native data platforms, ensure data quality and observability, and enable analytics across the organization.

The ideal candidate will own end-to-end data solutions, implement CDC with Debezium, and collaborate on architecture discussions while deploying via Docker and Kubernetes in a distributed environment.

Qualifications

• Strong experience in building scalable data pipelines (batch & streaming).
• Proficiency in Python, SQL, and data structures & algorithms.
• Hands-on with cloud platforms (AWS) and data lake architectures.
• Experience with CDC, Apache Airflow or Prefect, and containerized apps.

Responsibilities

• Design, build, and maintain scalable batch and real-time data pipelines.
• Develop streaming solutions using Kafka and Spark Structured Streaming.
• Implement CDC pipelines using Debezium.
• Build and manage Data Lake architectures on AWS.
• Orchestrate workflows using Airflow and Prefect.
• Write production-grade Python code with a focus on reliability.
• Deploy and manage apps using Docker and Kubernetes.
• Participate in HLD/LLD and system design discussions.

Skills

Python

Apache Kafka

Apache Spark

AWS

CDC using Debezium

Apache Airflow

Prefect

SQL

DSA

Data Lake architectures

Docker & Kubernetes

HLD/LLD and System Design

Streaming data pipelines

Education

B.Tech / B.E. in Computer Science / IT

Tools

Docker

Kubernetes

Debezium

Job description

Senior Data Engineer

Location: Gurgaon, Sector 32
Experience: 3-6 Years

Job Summary

We are seeking a highly skilled Senior Data Engineer with strong expertise in Python, Kafka, Spark, AWS, and distributed data systems. The ideal candidate will be responsible for building scalable batch and streaming data pipelines, designing cloud-native data platforms, and enabling reliable data solutions for analytics and business applications.

Key Responsibilities

• Design, build, and maintain scalable batch and real-time data pipelines.
• Develop streaming solutions using Kafka and Spark Structured Streaming.
• Implement CDC (Change Data Capture) pipelines using Debezium.
• Build and manage Data Lake architectures on AWS.
• Orchestrate workflows using Apache Airflow and Prefect.
• Write high-quality, production-grade code in Python.
• Ensure data quality, monitoring, observability, and reliability.
• Deploy and manage applications using Docker and Kubernetes.
• Participate in HLD/LLD, architecture discussions, and performance optimization.

Required Skills

• Strong expertise in Python
• Apache Kafka
• Apache Spark (Batch & Streaming)
• AWS (S3, EC2, IAM, EMR, Glue, EKS)
• CDC using Debezium
• Apache Airflow
• Prefect
• Strong SQL
• Data Structures & Algorithms (DSA)
• Data Lake architectures
• Docker & Kubernetes
• HLD/LLD and System Design
• Experience building streaming data pipelines

Good to Have

• Apache Superset or similar BI tools
• Data observability, monitoring, and alerting
• ML data pipelines and feature engineering
• Iceberg, Delta Lake, or Hudi

Education

B.Tech / B.E. in Computer Science, Information Technology, or a related engineering discipline.