Azure Data Engineering Lead - Pipelines, Databricks & Data Lake

Tata Consultancy Services

Stamford (CT)

On-site

USD 110,000 - 130,000

Full time

3 days ago

Be an early applicant

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Benefits offered by this job

Discretionary Annual Incentive

Medical Coverage

Family Support

401K Plan

Training Reimbursement

Job summary

Tata Consultancy Services in Stamford, CT seeks a Lead Azure Data Engineer to guide the data engineering team and own delivery across the Azure data platform. The role focuses on building scalable ETL/ELT pipelines, data transformations, and governance in a cloud-first environment.

You will design and implement metadata-driven ingestion, validation, and monitoring while migrating legacy workloads to modern data lake architectures. Strong PySpark and Databricks skills are essential.

Qualifications

Lead the Azure Data Engineering team and provide day-to-day technical direction.
Develop scalable data pipelines using ADF for batch ingestion and transformation.
Build reusable data-transformation frameworks with PySpark and Azure Databricks notebooks.
Design metadata-driven ingestion, transformation, validation, and restart components.

Responsibilities

Lead the Azure Data Engineering team and provide day-to-day technical direction, delivery oversight, and issue resolution.
Design, develop, orchestrate, and maintain scalable ADF pipelines for batch ingestion, data integration, transformation, and downstream processing.
Develop complex data-transformation frameworks using PySpark and Azure Databricks notebooks.
Migrate RDBMS and ETL workloads to modern Data Lake and ELT patterns.
Optimize PySpark workloads for performance and scalability.