Azure Databricks Data Engineer

Tata Consultancy Services; St. Louis (MO); On-site; USD 110,000 - 135,000; Full time

Benefits offered by this job

Discretionary Annual Incentive

Comprehensive Medical Coverage

Maternal & Parental Leaves

Auto & Home Insurance

Identity Theft Protection

Commuter Benefits

Certification & Training Reimbursement

Vacation, Time Off, Holidays

401K Plan

Performance Bonus

College Fund

Student Loan Refinancing

Pet Insurance Plans

Job summary

Tata Consultancy Services Limited in Saint Louis, MO, is hiring a data engineering specialist to design and build scalable batch data pipelines on Azure Databricks, delivering bronze-to-gold data structures and certified data products. You will implement quality checks, governance, and reusable HR data transformations.

The role requires deep SQL, Spark, and Delta Lake expertise, along with Unity Catalog integration and experience in an on-site, multinational delivery setup.

Qualifications

• Databricks platform and PySpark proficiency for large-scale batch pipelines.
• Advanced SQL development and performance tuning for analytics.
• Experience with Azure Databricks production-grade pipelines and lakehouse patterns.
• Hands-on with Delta Lake table design, schema evolution, and upsert patterns.
• Familiarity with Unity Catalog, data governance, and security controls.
• Experience in an Agile, onshore/offshore delivery model.

Responsibilities

• Design and develop scalable batch data pipelines in Azure Databricks using PySpark and Spark SQL.
• Build bronze, silver, and gold data structures and data products with quality checks.
• Translate business metrics into technical designs and reusable transformations for HR data.
• Implement data-quality checks, reconciliation, lineage, and documentation.
• Optimize workloads, Delta tables, partitions, and cluster usage for performance and cost.
• Ensure security/governance including Unity Catalog and row-level access patterns.
• Coordinate with ingestion, data governance, DevOps, and BI teams to define data contracts.
• Contribute to CI/CD, automated testing, deployment, and production-readiness activities.
• Mentor engineers and analysts, and explain technical decisions to stakeholders.

Skills

Databricks

Azure Databricks

PySpark

Spark SQL

Delta Lake

Medallion architecture

Data modeling

Data quality

Unit testing

CI/CD

Git

Unity Catalog

Informatica

Workday

Dayforce

PeopleSoft

Power BI

SQL

Delta Live Tables

Lakeflow

Data governance

Row-level security

Agile

Offshore delivery

Tools

Delta Live Tables

Databricks

Spark Declarative Pipelines

Unity Catalog

Informatica

Workday

Dayforce

PeopleSoft

Power BI

Azure Data Lake

Job description

Must Have Technical/Functional Skills

• Significant data engineering experience, including substantial hands‑on Databricks development
• experience.
• Azure Databricks - Strong hands‑on experience building production‑grade batch pipelines and
• lakehouse solutions.
• PySpark / Spark SQL - Advanced transformations, joins, window functions, reusable frameworks,
• debugging, and performance tuning.
• Delta Lake - Table design, schema evolution, merge/upsert patterns, optimization, data reliability,
• and history management.
• Medallion architecture - Practical delivery experience across source-aligned, conformed, and
• consumption-ready layers.
• Data modeling - Fact/dimension design, conformed dimensions, slowly changing dimensions,
• aggregates, and semantic-consumption considerations.
• Data quality and testing - Profiling, rule implementation, reconciliation, unit/integration tests,
• controls, and defect analysis.
• Azure ecosystem - Working knowledge of Azure Data Lake Storage and relevant Azure
• integration, identity, monitoring, and DevOps services.
• DevOps - Git-based development, code reviews, CI/CD, environment promotion, release controls,
• and automated deployment.
• Security/governance - Experience implementing least-privilege access and working with catalog,
• lineage, classification, and row-level security requirements.
• SQL - Advanced SQL development and query optimization for large analytical datasets.
• Hands‑on exposure to Databricks Lakeflow Spark Declarative Pipelines or Delta Live Tables and
• the ability to adopt evolving Databricks‑native pipeline standards.
• Experience with Unity Catalog; familiarity with enterprise data-governance and data-quality tooling
• such as Informatica.
• Experience with HR data from Workday, Dayforce, PeopleSoft, recruiting, workforce
• management, or talent systems.
• Understanding of Power BI consumption patterns, star schemas, refresh needs, aggregates, and
• performance requirements.
• Experience creating accelerators, reusable engineering templates, metadata-driven pipelines, or
• automated test frameworks.
• Databricks Data Engineer certification or equivalent cloud/data engineering credential.
• Experience in a blended onshore/offshore Agile delivery model.

Roles & Responsibilities

• Design and develop scalable batch data pipelines in Azure Databricks using PySpark and Spark
• SQL.
• Build and maintain bronze, silver, and gold data structures, including certified fact and dimension
• data products.
• Translate business metric definitions, calculation rules, filters, granularity, history, and refresh
• requirements into technical designs.
• Create reusable transformations for HR domains such as retention, hiring, transfers, promotions,
• reviews, hours, and scheduling.
• Implement data-quality checks, reconciliation, exception handling, auditability, lineage, and
• technical documentation.
• Optimize Spark workloads, Delta tables, joins, partitions, file sizes, and cluster usage for
• performance and cost.
• Apply enterprise security and governance standards, including Unity Catalog integration,
• permissions, sensitive-data handling, and support for row-level access patterns.
• Coordina te with ingestion, data governance, modeling, DevOps, and Power BI teams to define
• clear data contracts and acceptance criteria.
• Contribute to CI/CD, source control, automated testing, deployment, operational monitoring, and
• production-readiness activities.
• Participate in requirement workshops and explain technical choices clearly to business, HR
• analytics, and engineering stakeholders.
• Support knowledge transfer and mentor participating engineers or analysts

Salary Range-$110,000-$135,000 a year

• Discretionary Annual Incentive.
• Comprehensive Medical Coverage: Medical & Health, Dental & Vision, Disability Planning & Insurance, Pet Insurance Plans.
• Family Support: Maternal & Parental Leaves.
• Insurance Options: Auto & Home Insurance, Identity Theft Protection.
• Convenience & Professional Growth: Commuter Benefits & Certification & Training Reimbursement.a
• Time Off: Vacation, Time Off, Sick Leave & Holidays.
• Legal & Financial Assistance: Legal Assistance, 401K Plan, Performance Bonus, College Fund, Student Loan Refinancing

Saint Louis, MO

TECHNOLOGY

Engineer

435301

Azure

$110,000 - $135,000