Senior Data Engineer

MetAntz; Bengaluru; Hybrid; INR 4,000,000 - 8,500,000; Full time

Job summary

MetAntz is hiring for a senior data engineering role to design and operate real-time ETL/ELT pipelines using Apache Flink on Kubernetes. You will implement CDC workflows with Debezium for PostgreSQL, write Flink SQL/DataStream logic, and build connectors to StarRocks while ensuring fault-tolerant stateful processing.

The role requires 5+ years in data engineering, strong Java/Python/Scala skills, and hands-on experience with Flink on GKE.

Qualifications

• 5+ years in data engineering focused on ETL/ELT pipelines.
• 2+ years with Apache Flink in production.
• Strong Java, Scala, or Python skills for Flink jobs.
• Experience with CDC patterns (Flink CDC, Debezium).
• Hands-on PostgreSQL data source experience and StarRocks familiarity.

Responsibilities

• Design, build, and maintain high-performance real-time ETL/ELT pipelines with Flink.
• Architect CDC solutions using Flink CDC connectors and Debezium for PostgreSQL.
• Develop Flink SQL and DataStream pipelines for complex transformations.
• Create and maintain StarRocks integration connectors and sinks.
• Ensure fault-tolerance with checkpointing and state management strategies.

Skills

Flink

PostgreSQL

StarRocks

Debezium

Flink SQL

DataStream API

CDC

Java

Python

Scala

Kafka

Kubernetes

GKE

Helm

Harness CI/CD

Airflow

Prometheus

Grafana

StarRocks Connector

Flink on Kubernetes

Tools

Flink environment tooling

PostgreSQL

StarRocks Connector

Debezium

Kubernetes (GKE)

Helm

Harness CI/CD

Prometheus

Grafana

Apache Airflow

Job description

About MetAntz

MetAntz has one of the most advanced GenAI recruitment platforms. We help companies hire faster with an end-to-end product that covers job posting, sourcing, scoring and matching, and AI-driven assessments. We also run a candidate marketplace and a Recruitment as a Service team that fills roles for our customers. In this role you will sell both the platform and the services around it.

About MetAntz

MetAntz has one of the most advanced GenAI recruitment platforms. We help companies hire faster with an end-to-end product that covers job posting, sourcing, scoring and matching, and AI-driven assessments. We also run a candidate marketplace and a Recruitment as a Service team that fills roles for our customers. In this role you will sell both the platform and the services around it.

Key Responsibilities

Pipeline Development and Architecture

Qualifications And Experience

Technical Skills (Must Have)

Preferred Qualifications

• Design, build, and maintain high-performance Apache Flink ETL,ELT pipelines for real-time data synchronisation from PostgreSQL to StarRocks.
• Architect robust CDC (Change Data Capture) solutions using Flink CDC connectors and Debezium for PostgreSQL source ingestion.
• Implement Flink SQL and DataStream API pipelines for complex transformation logic, aggregations, and data enrichment.
• Develop and maintain custom Flink connectors and sinks for StarRocks integration using the StarRocks Flink Connector.
• Design fault-tolerant, exactly-once or at-least-once pipelines with appropriate checkpointing and state management strategies.
• Evaluate and implement schema evolution strategies to handle upstream PostgreSQL schema changes gracefully. Performance Optimisation
• Profile and tune Flink job performance parallelism settings, task manager memory, operator chaining, and back-pressure management.
• Optimise StarRocks loading strategies (Stream Load vs. Routine Load) for high-throughput ingestion.
• Monitor pipeline latency and throughput SLAs; proactively identify and resolve bottlenecks.
• Implement efficient watermarking and windowing strategies for time-sensitive data flows.
• Manage Flink state backends (RocksDB , heap) and configure appropriate TTLs to control state size. Troubleshooting and Reliability
• Own end-to-end pipeline reliability diagnose and resolve issues including data lag, job failures, checkpoint timeouts, and OOM errors.
• Establish alerting and observability for pipeline health using Flink metrics, Prometheus, and Grafana (or equivalent).
• Define and implement data quality checks, reconciliation processes, and dead-letter queue (DLQ) strategies.
• Perform root-cause analysis on data discrepancies between PostgreSQL source and StarRocks target.
• Maintain comprehensive runbooks for common failure scenarios and recovery procedures.
• Team Leadership and Delivery
• Lead Data Engineers assign tasks, conduct code reviews, and ensure delivery against sprint goals.
• Mentor engineers on Flink internals, best practices, and performance considerations.
• Collaborate with data consumers (analysts, BI teams) to understand requirements and translate them into pipeline specifications.
• Drive technical decisions on tooling, frameworks, and deployment strategies (Flink on Kubernetes on GCP).
• Maintain technical documentation including architecture diagrams, data flow documentation, and operational guides.
• Deployment and DevOps
• Manage Flink cluster deployment and configuration on Kubernetes (GKE) using Helm charts and Harness CI,CD pipelines.
• Build and maintain CI,CD pipelines using Harness for Flink job packaging, testing, and deployment to GKE.
• Manage Flink cluster configuration using Helm charts; maintain Helm values and chart templates for environment-specific configurations.
• Coordinate with infrastructure and DBA teams for PostgreSQL slot management and StarRocks table design.
• 5+ years of experience in data engineering with a focus on ETL,ELT pipeline development.
• 2+ years of hands-on production experience with Apache Flink (Flink SQL and,or DataStream API).
• Strong proficiency in Java, Scala, or Python for Flink job development.
• Experience with Change Data Capture (CDC) patterns Flink CDC, Debezium, or equivalent.
• Practical experience with PostgreSQL as a data source, including replication slots and WAL configuration.
• Demonstrated experience with columnar OLAP databases (StarRocks, Doris, ClickHouse, or similar).
• Solid understanding of distributed systems concepts fault tolerance, exactly-once semantics, state management, and watermarking.
• Experience with Kafka or similar message brokers as part of streaming architectures.
• Proven ability to lead engineering teams, including task planning, code review, and mentoring.
• Strong analytical and problem-solving skills with the ability to debug complex distributed pipeline issues.
• Excellent written and verbal communication skills with the ability to document technical decisions clearly.
• Self-driven with the ability to work autonomously and manage priorities in a fast-paced environment.
• Hands-on experience with StarRocks Flink Connector and StarRocks primary-key table designs.
• Hands-on experience with Flink on Kubernetes (GKE) using Flink Kubernetes Operator or native K8s mode.
• Experience with Harness CI,CD and Helm chart management for data workloads.
• Experience with Flink Table API and Flink SQL for unified batch and stream processing.
• Knowledge of Apache Iceberg, Delta Lake, or Hudi for lakehouse architectures.
• Experience with orchestration tools such as Apache Airflow or Prefect for hybrid batch,streaming workflows.
• Familiarity with observability stacks Prometheus, Grafana, and Flink metrics reporters.
• Prior experience in a technical lead or senior engineer role with delivery accountability.
• Understanding of data modelling best practices for analytical workloads in StarRocks. This is a hybrid role based in Bangalore, with an expectation to work from the office three days a week.