Data Engineer

moglix; Dadri; On-site; INR 1,200,000 - 2,400,000; Full time

Job summary

moglix is seeking a data engineering professional to build, test, and maintain batch and streaming data pipelines on a Kafka-based platform. You will translate component designs into production-ready code and work across teams to enable reliable data movement from multiple sources into BigQuery and the data lake.

You will set up Kafka Connect connectors, monitor pipeline health, and implement ETL jobs to produce business-ready tables for downstream consumption.

Responsibilities

• Build, test and maintain batch and streaming data pipelines using Kafka, Spark and SQL.
• } ,

Skills

Kafka

Spark

SQL

BigQuery

ETL pipelines

Data governance

Monitoring

Tools

Apache Kafka

Kafka Connect

Debezium CDC

Schema Registry

Spark Structured Streaming

Databricks

GCP BigQuery

Airflow

Job description

Role & responsibilities

The person will work on our Kafka platform (Confluent/Apache Kafka, Kafka Connect, Debezium CDC, Schema Registry) and on pipelines that land data from MySQL, SQL Server, SAP HANA and MongoDB into BigQuery and the data lake. They will take component designs from senior engineers and turn them into tested, production-ready code.

Key responsibilities

1. Build, test and maintain batch and streaming data pipelines using Kafka, Spark and SQL.
2. Set up and operate Kafka Connect source and sink connectors (Debezium CDC, BigQuery sink, MongoDB, JDBC), including configs, SMTs, converters and Schema Registry.
3. Monitor pipeline health: consumer lag, connector and task failures, dead letter queues. Fix what you can and escal… alerts that need senior help.
4. Create ETL jobs that transform raw data into business-ready tables for downstream consumers.
5. Implement component designs given by senior engineers, following our established patterns, coding guidelines and best practices.
6. Write unit tests and take part in code reviews, both giving and receiving.
7. Work with Data Architects, Data Analysts and business teams to turn requirements into technical solutions, and keep data accurate and accessible.
8. Follow data governance and security practices: access control, schema evolution rules, data quality checks.

Good-to-have skills

• Kafka KRaft mode, multi-cluster setups or disaster recovery (MirrorMaker 2, Cluster Linking)
• Kafka Streams or ksqlDB
• Databricks, Deta Lake or Spark Structured Streaming
• Monitoring with Prometheus, Grafana or Kafka UI tools
• Orchestration with Airflow or a similar scheduler
• GCP (BigQuery IAM, GCS) or another cloud
• Data governance tooling: data catalogs, lineage, data quality checks