Azure Databricks Data Engineer: Batch Pipelines & Lakehouse

Tata Consultancy Services Limited; St. Louis (MO); On-site; USD 110,000 - 135,000; Full time

Benefits offered by this job

Discretionary Annual Incentive

Comprehensive Medical Coverage

Maternal & Parental Leaves

Auto & Home Insurance

Identity Theft Protection

Commuter Benefits

Certification & Training Reimbursement

Vacation, Time Off, Holidays

401K Plan

Performance Bonus

College Fund

Student Loan Refinancing

Pet Insurance Plans

Job summary

Tata Consultancy Services Limited in Saint Louis, MO, is hiring a data engineering specialist to design and build scalable batch data pipelines on Azure Databricks, delivering bronze-to-gold data structures and certified data products. You will implement quality checks, governance, and reusable HR data transformations.

The role requires deep SQL, Spark, and Delta Lake expertise, along with Unity Catalog integration and experience in an on-site, multinational delivery setup.

Qualifications

• Databricks platform and PySpark proficiency for large-scale batch pipelines.
• Advanced SQL development and performance tuning for analytics.
• Experience with Azure Databricks production-grade pipelines and lakehouse patterns.
• Hands-on with Delta Lake table design, schema evolution, and upsert patterns.
• Familiarity with Unity Catalog, data governance, and security controls.
• Experience in an Agile, onshore/offshore delivery model.

Responsibilities

• Design and develop scalable batch data pipelines in Azure Databricks using PySpark and Spark SQL.
• Build bronze, silver, and gold data structures and data products with quality checks.
• Translate business metrics into technical designs and reusable transformations for HR data.
• Implement data-quality checks, reconciliation, lineage, and documentation.
• Optimize workloads, Delta tables, partitions, and cluster usage for performance and cost.
• Ensure security/governance including Unity Catalog and row-level access patterns.
• Coordinate with ingestion, data governance, DevOps, and BI teams to define data contracts.
• Contribute to CI/CD, automated testing, deployment, and production-readiness activities.
• Mentor engineers and analysts, and explain technical decisions to stakeholders.

Skills

Databricks

Azure Databricks

PySpark

Spark SQL

Delta Lake

Medallion architecture

Data modeling

Data quality

Unit testing

CI/CD

Git

Unity Catalog

Informatica

Workday

Dayforce

PeopleSoft

Power BI

SQL

Delta Live Tables

Lakeflow

Data governance

Row-level security

Agile

Offshore delivery

Tools

Delta Live Tables

Databricks

Spark Declarative Pipelines

Unity Catalog

Informatica

Workday

Dayforce

PeopleSoft

Power BI

Azure Data Lake

Job description

Tata Consultancy Services Limited in Saint Louis, MO, is hiring a data engineering specialist to design and build scalable batch data pipelines on Azure Databricks, delivering bronze-to-gold data structures and certified data products. You will implement quality checks, governance, and reusable HR data transformations.

The role requires deep SQL, Spark, and Delta Lake expertise, along with Unity Catalog integration and experience in an on-site, multinational delivery setup.