Data Engineering manager

Worldclass Tech Talent Pvt. Ltd.; Dadri; On-site; INR 4,200,000 - 6,400,000; Full time

Job summary

Worldclass Tech Talent Pvt. Ltd. in Noida seeks a Data Engineering Manager with 10-15 years' experience to lead streaming and analytics-ready data pipelines. You will own the roadmap, guide real-time architectures, and collaborate with ML, product, and business teams to drive insights.

Role requires strong Java/Python tooling and hands-on Kafka, Pinot, Redshift, and AWS skills to build scalable data lakes on S3. Mentoring and governance are key parts of the role.

Qualifications

• Proven expertise in Kafka/MSK, Debezium, and real-time event-driven architectures.
• Hands-on with Pinot, Redshift, RocksDB or NoSQL DB.
• Strong background in custom tooling using Java and Python.
• Experience with Apache Airflow, Superset, Iceberg, Athena, and Glue.
• AWS ecosystem knowledge (IAM, S3, Lambda, Glue, ECS/EKS, CloudWatch).
• Deep understanding of data lake architecture, streaming vs batch processing, CDC concepts.
• Familiarity with Parquet, Avro formats and storage abstractions.

Responsibilities

• Own the data engineering roadmap from streaming ingestion to analytics-ready layers.
• Lead and evolve real-time data pipelines using Kafka/MSK, DebeziumCDC, Pinot.
• Design and maintain Java/Python-based data ingestion, validation, transformation, and replication frameworks.
• Design and manage data lakes using Iceberg, Glue, and Athena on S3.
• Define data modelling standards across Pinot, Redshift and warehouse layers.
• Lead open-source tools and orchestration platforms (Airflow).
• Drive cloud-native deployment strategies (ECS/EKS, Terraform/CDK, Docker).
• Collaborate with ML, product, and business teams to support analytics use cases.
• Mentor data engineers and promote best practices in architecture.
• Ensure observability, lineage, data quality, and governance across stack.

Job description

Job Title:Data Engineering manager
Location: Noida
Job Type:Full-time

Job Overview:

We are looking for a highly skilled and motivated Data engineering manager with 10-15 years of experience to join our growing team. You will create data pipelines to move data from source to data lake, Maintain the data lake, real time transform the data to uncover actionable insights and contribute to strategic decision-making across various departments. The ideal candidate will have strong analytics and programming skills, a solid understanding of data pipeline and programming logic, and the ability to collaborate effectively with cross-functional teams. You will be helping all application teams by providing data interface, also will be ensuring that data analytics team is getting required data for their initiatives.

Responsibilities

• Own the data engineering roadmap from streaming ingestion to analytics-ready layers.
• Lead and evolve our real-time data pipelines using Apache Kafka/MSK, DebeziumCDC, and Apache Pinot.
• Design and maintain custom-built Java/Python-based frameworks for data ingestion, validation, transformation, and replication.
• Design and manage data lakes using Apache Iceberg, Glue, and Athena over S3.
• Define and enforce data modelling standards across Pinot, Redshift and warehouse layers.
• Lead implementation and scaling of open-source tools and orchestration platforms (Airflow).
• Drive cloud-native deployment strategies using ECS/EKS, Terraform/CDK, Docker.
• Collaborate with ML, product, and business teams to support advanced analytics and AI use cases.
• Mentor data engineers and evangelize modern architecture and engineering best practices.
• Ensure observability, lineage, data quality, and governance across the data stack.

Technical Skills & Expertise

Must-Have:

• Proven expertise in Kafka/MSK, Debezium, and real-time event-driven architectures
• Hands-on experience with Pinot, Redshift, RocksDB or noSQL DB
• Strong background in custom tooling using Java and Python
• Experience with Apache Airflow, Superset, Iceberg, Athena, and Glue
• Strong AWS ecosystem knowledge (IAM, S3, Lambda, Glue, ECS/EKS, CloudWatch, etc.)
• Deep understanding of data lake architecture, streaming vs batch processing, CDCconcepts
• Familiarity with modern data formats (Parquet, Avro) and storage abstractions

Nice-to-Have:

• Exposure to dbt, Trino/Presto, ClickHouse, or Druid
• Familiarity with data security practices, encryption at rest/in-transit, GDPR/PCI compliance
• Experience with DevOps practices, GitHub Actions, CI/CD, and Terraform/CloudFormationRole & responsibilities