Senior Databricks Data Engineer

Tata Consultancy Services; New York (NY); On-site; USD 110,000 - 120,000; Full time

Benefits offered by this job

Discretionary Annual Incentive

Medical Coverage

Parental Leaves

401K Plan

Job summary

Tata Consultancy Services Limited in New York City seeks an Engineer to design and operate data pipelines using Databricks, PySpark and Delta Lake. The role focuses on ingestion, transformation, and reliable production processing in a fast-paced tech environment.

You will implement batch and streaming workflows, automate deployments via CI/CD, and collaborate with offshore/on-site teams. Strong communication and problem-solving are essential.

Qualifications

• Experience designing and building production Databricks ingestion pipelines.
• Strong knowledge of Delta Lake optimization and streaming/batch processing.
• Ability to automate deployment via CI/CD and manage data quality and governance.

Responsibilities

• Design and build production Databricks ingestion and transformation pipelines.
• Implement batch/streaming processing and Delta Lake optimization.
• Automate deployment and scheduling through CI/CD and workflows.
• Monitor, reconcile and troubleshoot data pipelines.

Skills

Databricks

PySpark

Spark SQL

Delta Lake

ETL/ELT

Data validation

Production support

CI/CD

MLOps

Education

Bachelor's degree in Computer Science

Tools

Auto Loader

Structured Streaming

AWS S3

Git

Workflow orchestration

Performance tuning

Job description

Job Description

Must Have Technical/Functional Skills

• Databricks, PySpark, Spark SQL and Delta Lake
• Auto Loader, Structured Streaming, AWS S3 and ETL/ELT
• Workflow orchestration, Git/CI/CD and performance tuning
• Data validation, reconciliation and production support

Roles & Responsibilities

• Design and build production Databricks ingestion and transformation pipelines
• Implement batch/streaming processing and Delta Lake optimization
• Automate deployment and scheduling through CI/CD and workflows
• Monitor, reconcile and troubleshoot data pipelines

In addition, should be able to do the following

• Play the role of onsite co-Ordinator for offshore
• Able to conduct RCA and provide and Implement solutions

Generic Managerial Skills, If any

• Customer-facing communication and concise executive reporting
• Onsite-offshore coordination across distributed teams
• Structured ownership, prioritization, risk escalation and collaboration
• Ability to work independently while following enterprise governance and security standards

Salary Range

$110,000-$120,000 a year

TCS Employee Benefits Summary:

• Discretionary Annual Incentive.
• Comprehensive Medical Coverage: Medical & Health, Dental & Vision, Disability Planning & Insurance, Pet Insurance Plans.
• Family Support: Maternal & Parental Leaves.
• Insurance Options: Auto & Home Insurance, Identity Theft Protection.
• Convenience & Professional Growth: Commuter Benefits & Certification & Training Reimbursement.
• Time Off: Vacation, Time Off, Sick Leave & Holidays.
• Legal & Financial Assistance: Legal Assistance, 401K Plan, Performance Bonus, College Fund, Student Loan Refinancing.

Location

new york city, NY

Function

TECHNOLOGY

Role

Engineer

ID

437619

Skills

Spark | SQL

Salary Range

$110,000 - $120,000

Desired Candidate Profile

Qualifications

BACHELOR OF COMPUTER SCIENCE