Databricks Data Engineer - Production Pipelines Delta Lake

Tata Consultancy Services Limited; New York (NY); On-site; USD 110,000 - 120,000; Full time

Benefits offered by this job

Discretionary Annual Incentive

Medical Coverage

Parental Leaves

401K Plan

Job summary

Tata Consultancy Services Limited in New York City seeks an Engineer to design and operate data pipelines using Databricks, PySpark and Delta Lake. The role focuses on ingestion, transformation, and reliable production processing in a fast-paced tech environment.

You will implement batch and streaming workflows, automate deployments via CI/CD, and collaborate with offshore/on-site teams. Strong communication and problem-solving are essential.

Qualifications

• Experience designing and building production Databricks ingestion pipelines.
• Strong knowledge of Delta Lake optimization and streaming/batch processing.
• Ability to automate deployment via CI/CD and manage data quality and governance.

Responsibilities

• Design and build production Databricks ingestion and transformation pipelines.
• Implement batch/streaming processing and Delta Lake optimization.
• Automate deployment and scheduling through CI/CD and workflows.
• Monitor, reconcile and troubleshoot data pipelines.

Skills

Databricks

PySpark

Spark SQL

Delta Lake

ETL/ELT

Data validation

Production support

CI/CD

MLOps

Education

Bachelor's degree in Computer Science

Tools

Auto Loader

Structured Streaming

AWS S3

Git

Workflow orchestration

Performance tuning

Job description

Tata Consultancy Services Limited in New York City seeks an Engineer to design and operate data pipelines using Databricks, PySpark and Delta Lake. The role focuses on ingestion, transformation, and reliable production processing in a fast-paced tech environment.

You will implement batch and streaming workflows, automate deployments via CI/CD, and collaborate with offshore/on-site teams. Strong communication and problem-solving are essential.