Lead Data Platform Engineer

Tata Consultancy Services; Hyderabad; On-site; INR 3,500,000 - 7,000,000; Full time

Job summary

Tata Consultancy Services seeks a senior data engineer with 8-15 years of experience to design and optimize distributed data platforms using Spark, Kafka, Flink, and Airflow on GKE. You will implement Iceberg tables, manage Nessie catalogs, and ensure scalable performance across multi-tenant environments.

You will mentor teams, drive best practices, and collaborate with stakeholders to deliver reliable data pipelines, governance, and cost-aware cloud solutions.

Qualifications

• 8-15 years of data engineering or data platform experience.
• Hands-on Apache Spark with DataFrame/SQL workloads, tuning, and production troubleshooting.
• Strong Apache Kafka experience covering topics, partitions, consumer groups, offsets, delivery semantics, schemas, security, and operations.
• Practical Apache Flink experience covering state, checkpoints, savepoints, event time, recovery, and backpressure.
• Strong understanding of Apache Iceberg, GCS, schema/partition evolution, snapshots, compaction, and table maintenance.
• Experience with Apache Polaris, Nessie, or REST-based Iceberg catalogs.
• Hands-on experience with Apache Airflow for distributed data-processing orchestration.
• Working knowledge of GKE-based execution, containers, Helm, service accounts, Workload Identity Federation, secrets, networking, storage, and observability.
• Proficiency in Python, Java, or Scala, SQL, Git, automated testing, CI/CD, and reusable software engineering practices.
• Strong technical leadership, troubleshooting, documentation, mentoring, and stakeholder communication skills.

Responsibilities

Skills

Spark

Kafka

Flink

Airflow

Iceberg

Polaris/Nessie

Kubernetes

Python

Java

Scala

SQL

Git

CI/CD

Automated testing

Leadership

Tools

GKE

Helm

Containers

Job description

Required Skill set- Spark, Kafka, Flink, Airflow, Iceberg, Polaris/Nessie, Kubernetes

Required Skills & Experience

Must Have

• 8-15 years of data engineering or data platform experience, with substantial recent hands-on responsibility for distributed processing platforms.
• Deep hands-on experience with Apache Spark, including DataFrame or SQL workloads, performance tuning, packaging, configuration, and production troubleshooting.
• Strong Apache Kafka experience covering topics, partitions, consumer groups, offsets, delivery semantics, schemas, security, and operations.
• Practical Apache Flink experience covering state, checkpoints, savepoints, event time, recovery, and backpressure.
• Strong understanding of Apache Iceberg, Google Cloud Storage, schema and partition evolution, snapshots, compaction, and table maintenance.
• Experience with Apache Polaris, Project Nessie, or REST-based Iceberg catalogs.
• Hands-on experience with Apache Airflow for distributed data-processing orchestration.
• Working knowledge of GKE-based execution, containers, Helm, service accounts, Workload Identity Federation, secrets, networking, storage, and observability.
• Proficiency in Python, Java, or Scala, SQL, Git, automated testing, CI/CD, and reusable software engineering practices.
• Strong technical leadership, troubleshooting, documentation, mentoring, and stakeholder communication skills.

Good to Have

• Experience with Trino, OpenMetadata, JupyterHub, Dataproc, data-quality frameworks, lineage, or schema registries.
• Exposure to Debezium, Kafka Connect, Datastream, database logs, or comparable CDC patterns.
• Experience with Spark Operator, Flink Kubernetes Operator, Strimzi, KubernetesExecutor, or operator-based GKE integration.
• Understanding of data security, encryption, masking, governance, data contracts, and regulated enterprise controls.

Experience with multi-tenant data platforms, regulated environments, and platform cost optimisation on GCP.