Databricks Platform Engineer — Unity Catalog & DataOps

Tata Consultancy Services Limited; Irving (TX); On-site; USD 125,000 - 140,000; Full time

Job summary

Tata Consultancy Services Limited seeks a seasoned Data Engineer to design and operate batch data pipelines on Spark within AWS and Databricks environments. You will own Iceberg table design, Unity Catalog governance, and robust data loading frameworks, working across NAS, S3, and OpenShift-based loading layers.

In the role you will monitor pipelines, implement incident runbooks, and collaborate with Data Stewards and DBAs to ensure data quality, security, and compliant data practices while

Qualifications

• Overall 10 years' experience; 4+ years of data engineering with 2+ years Spark in production.
• Solid cloud networking fundamentals (VPC, subnets, endpoints) and AWS services (S3, EKS, IAM, KMS, SQS, MSK).
• Production experience with Iceberg or Delta Lake, and familiarity with Kubernetes concepts.
• Proficient Python and SQL; understanding IAM, STS, and credential management.

Responsibilities

• Design and build batch data pipelines using Spark on EKS and Databricks.
• Own Bronze-Silver-Gold data pipeline architecture for assigned domains.
• Write and maintain Iceberg table DDL following naming standards.
• Integrate with loading frameworks (Autosys, OpenShift, Airflow) and evolve them.
• Monitor pipeline health, incident runbooks, and on-call rotations.
• Collaborate with Data Stewards and DBAs on schema governance and data quality.
• Participate in platform governance uplift including UC volumes migration.

Skills

Apache Spark

Python

SQL

AWS

Kubernetes

Education

Bachelor's degree in Computer Science

Tools

Iceberg

Delta Lake

Unity Catalog

Databricks

Airflow

Job description

Tata Consultancy Services Limited seeks a seasoned Data Engineer to design and operate batch data pipelines on Spark within AWS and Databricks environments. You will own Iceberg table design, Unity Catalog governance, and robust data loading frameworks, working across NAS, S3, and OpenShift-based loading layers.

In the role you will monitor pipelines, implement incident runbooks, and collaborate with Data Stewards and DBAs to ensure data quality, security, and compliant data practices while