Databricks Platform Engineer

Tata Consultancy Services; Irving (TX); On-site; USD 125,000 - 140,000; Full time

Job summary

Tata Consultancy Services Limited seeks a seasoned Data Engineer to design and operate batch data pipelines on Spark within AWS and Databricks environments. You will own Iceberg table design, Unity Catalog governance, and robust data loading frameworks, working across NAS, S3, and OpenShift-based loading layers.

In the role you will monitor pipelines, implement incident runbooks, and collaborate with Data Stewards and DBAs to ensure data quality, security, and compliant data practices while

Qualifications

• Overall 10 years' experience; 4+ years of data engineering with 2+ years Spark in production.
• Solid cloud networking fundamentals (VPC, subnets, endpoints) and AWS services (S3, EKS, IAM, KMS, SQS, MSK).
• Production experience with Iceberg or Delta Lake, and familiarity with Kubernetes concepts.
• Proficient Python and SQL; understanding IAM, STS, and credential management.

Responsibilities

• Design and build batch data pipelines using Spark on EKS and Databricks.
• Own Bronze-Silver-Gold data pipeline architecture for assigned domains.
• Write and maintain Iceberg table DDL following naming standards.
• Integrate with loading frameworks (Autosys, OpenShift, Airflow) and evolve them.
• Monitor pipeline health, incident runbooks, and on-call rotations.
• Collaborate with Data Stewards and DBAs on schema governance and data quality.
• Participate in platform governance uplift including UC volumes migration.

Skills

Apache Spark

Python

SQL

AWS

Kubernetes

Education

Bachelor's degree in Computer Science

Tools

Iceberg

Delta Lake

Unity Catalog

Databricks

Airflow

Job description

Job Description

Data Engineering Responsibilities

• Design and build batch data pipelines using Apache Spark on EKS and Databricks, processing data from NAS and upstream source systems into Iceberg tables on S3
• Own the Bronze Silver Gold DLT pipeline architecture for assigned business domains
• Write and maintain DDL for Iceberg tables following the CDP naming standard (catalog.schema.table with region, LOB, and layer components)
• Integrate with existing loading framework (Autosys OpenShift S3 Airflow DAG trigger chain) and extend it as the platform evolves
• Work with Data Stewards and DBA team to ensure schema design meets data classification requirements (INT, PII, CPII, CSPII)
• Implement and validate Unity Catalog governance controls - column masking, row-level filters, and lineage tracking - for tables under your ownership
• Contribute to the migration of pipeline artifacts from DBFS to UC Volumes as part of the platform's governance uplift programmer

DataOps Responsibilities

• Own the operational health of pipelines in your domain - monitoring, alerting, failure triage, and restart procedures
• Build and maintain job monitoring integration between Databricks / Airflow and the team's observability stack (Splunk, CloudWatch, MSK audit events)
• Define and document incident runbooks for common failure patterns: EKS resource exhaustion, S3 VPC Endpoint connectivity, Spark executor out-of-memory, DLT pipeline schema evolution failures
• Manage deployment of pipeline code and DAG updates through the Lightspeed CI/CD pipeline - understand what goes to S3 config bucket, what goes to the DAG bucket, and what goes to EFS
• Participate in on-call rotation for production pipeline incidents
• Work with the platform team on capacity planning for EKS cluster - there is no YARN queue here, resource exhaustion means job failure
• Maintain data quality monitoring and reconciliation between on-prem and cloud outputs during the migration period

What We Are Looking For

Must Have

• Overall 10 years' experience; 4+ years of data engineering experience, with at least 2 years working with Spark in a production environment
• Solid understanding of cloud networking fundamentals - VPC, subnets, VPC Endpoints, Security Groups. You do not need to be a network engineer, but you need to understand why a Spark pod's network path matters and what breaks when it is wrong
• Hands-on experience with AWS services: S3, EKS, IAM, KMS, SQS, MSK
• Experience with Apache Iceberg or Delta Lake in production - not just awareness, but actual table design and maintenance
• Familiarity with Kubernetes - you do not need to administer a cluster, but you need to understand pod l ifecycle, resource limits, and how to read a pod describe output when something fails
• Comfortable reading and writing Python and SQL at a production standard
• Understanding of IAM and credential management - why static keys are unacceptable, how STS AssumeRole works, what Vault does

Strong Preference

• Experience with Databricks, specifically Unity Catalog, MLflow, and DLT
• Experience working in a regulated financial services environment - you understand why data classification, audit trails, and access controls are not optional extras
• Familiarity with Airflow (DAG design, KubernetesPodOperator, trigger chain design)
• Experience with Spark performance tuning - you know what data skew looks like, when to add executors vs CPU, and how to read a Spark history server
• Exposure to model governance or MLOps - understanding how MLflow Model Registry, Champion/Challenger alias management, and MRM review processes fit together

Not Required But Noticed

• Experience with Lakebase or other OLTP-on-lakehouse technologies
• Familiarity with Collibra or similar enterprise data governance platforms
• Experience designing UC group naming conventions or RBAC frameworks at scale

What You Will Not Be Doing

• Building dashboards or BI reports
• Managing the Databricks platform itself - that sits with the platform team

The Environment

• Compute: EKS (Spark), OpenShift (on-prem loading framework), Databricks (UC, MLflow, DLT)
• Storage: S3 (Iceberg), EFS (intermediate artifacts), NAS (source files)
• Orchestration: Airflow (MWA), Autosys (on-prem, being phased out)
• Messaging: MSK (Kafka), SQS, SNS
• Catalog: HMS on Aurora migrating to AWS Glue Unity Catalog
• Security: Vault (credential injection), KMS (encryption), PrivateLink (no public internet for data paths), CloudTrail (audit)
• CI/CD: Lightspeed (Citi internal), Bitbucket / GitHub

What Good Looks Like in This Role

• In the first 90 days, you should be able to: Explain the end-to-end data flow from NAS to an Iceberg table in S3, naming every component the data touches and the network path it takes
• Run a Spark job on EKS independently, debug a failure from the pod logs, and know when to look at the Spark History Server vs CloudTrail
• Identify a governance gap in a UC table configuration and propose a fix
• In six months, you should own at least one business domain end-to-end - pipeline design, operational health, schema governance, and incident response.

Salary Range: $125,000 to $140,000 per year

Location

Irving, TX

Function

TECHNOLOGY

Role

Engineer

ID

435866

Skills

Apache Spark

Salary Range

$125,000 - $140,000

Desired Candidate Profile

Qualifications

BACHELOR OF COMPUTER SCIENCE