Senior Data Engineer

Tata Consultancy Services; Bengaluru; On-site; INR 1,500,000 - 2,000,000; Full time

Job summary

Tata Consultancy Services is seeking a senior data engineer with 7–12 years of experience to own distributed batch and streaming pipelines. You will work with Spark (PySpark/Scala) and Spark SQL, tuning for production reliability and performance.

Role includes building lakehouse pipelines on Google Cloud Storage with Iceberg, strong SQL skills for profiling and transformations, and ingestion using Kafka Connect or Debezium. Collaboration with stakeholders is essential.

Qualifications

• 7–12 years data engineering with hands-on pipeline ownership.
• Strong Spark experience via PySpark/Scala/Spark SQL and production tuning.
• Lakehouse pipelines on Google Cloud Storage with Iceberg or similar.
• Strong SQL with profiling, transforms, joins, aggregations, and reconciliation.
• Experience with Kafka and Flink or Spark Structured Streaming.
• Airflow for scheduling, retries, backfills, and monitoring.
• Data ingestion using Kafka Connect, Debezium, Datastream, or equivalents.
• Understanding of data modeling, schema evolution, and governance.

Responsibilities

• Build and maintain distributed batch/streaming data pipelines.
• Tune and troubleshoot Spark jobs in production.
• Design lakehouse architectures on GCS using Iceberg or equivalent.
• Ensure data quality, lineage, and governance across datasets.

Skills

Distributed pipelines

Spark & Spark SQL

SQL proficiency

Kafka & streaming

Airflow

Data modeling & governance

Git & CI/CD

GKE & cloud infra

Problem solving & comms

Tools

Apache Spark

Iceberg

Google Cloud Storage

Kafka Connect / Debezium / Datastream

Flink / Structured Streaming

GKE

Secret Manager

Dataproc

Airflow

OpenMetadata / Trino / Nessie

Schema registries

Job description

• 7-12 years of data engineering experience, with strong recent hands‑on responsibility for distributed batch or streaming pipelines.
• Strong hands‑on Apache Spark experience using PySpark, Scala, or Spark SQL, including tuning and production troubleshooting.
• Experience building lakehouse pipelines on Google Cloud Storage using Apache Iceberg or a comparable open table format.
• Strong SQL skills with practical experience in profiling, transformations, joins, aggregations, incremental processing, and reconciliation.
• Experience with Apache Kafka and Apache Flink or Spark Structured Streaming.
• Hands‑on Apache Airflow experience covering scheduling, dependencies, retries, backfills, and monitoring.
• Experience with database, file, API, event, and CDC ingestion using Kafka Connect, Debezium, Datastream, or comparable tools.
• Understanding of data modeling, schema evolution, data contracts, partitioning, file formats, data quality, lineage, and governance.
• Working knowledge of Git, automated testing, CI/CD, containers, GKE‑based execution, Cloud Storage, Workload Identity Federation, and Secret Manager.
• Strong problem solving, documentation, collaboration, and stakeholder communication skills.

Good to Have

• Experience with Polaris, Nessie, REST catalogs, Trino, OpenMetadata, schema registries, Dataproc, or data‑quality frameworks.
• Experience with GCP data services and regulated enterprise environments.
• Exposure to banking data, high‑volume transaction processing, reconciliation, auditability, or data privacy controls.
• Experience with reusable data‑product patterns, data contracts, semantic models, or domain‑oriented data platforms.
• Exposure to performance engineering, recovery testing, and multi‑tenant GCP data platforms.