Senior Big Data Engineer - Databricks Lakehouse & Delta

Tata Consultancy Services; Plano (TX); On-site; USD 110,000 - 125,000; Full time

Benefits offered by this job

Discretionary Annual Incentive

Comprehensive Medical Coverage

Family Support: Maternal & Parental Le

Insurance Options: Auto & Home

Certification & Training Reimbursement

Time Off: Vacation & Holidays

Legal & Financial Assistance

401K Plan

Job summary

Tata Consultancy Services in the United States is seeking a Senior Big Data Engineer with 10-13 years of hands-on experience, specializing in Hadoop, PySpark, Kafka, Hive, and Databricks Lakehouse. The role focuses on building scalable, well-governed data platforms for regulatory, risk, trading, and analytics workloads, with Delta Lake, Bronze/Silver/Gold modeling, and real-time ingestion.

On-prem and cloud deployments, collaboration with quants and risk teams, and leadership of junior engineers

Qualifications

• 10-13 years of hands-on experience in Big Data engineering.
• Expertise in PySpark, Kafka, Hive, and Databricks Lakehouse architectures.
• Strong knowledge of Delta Lake, data modeling, and governance practices.
• Experience with on-prem and cloud big data platforms and CI/CD pipelines.

Responsibilities

• Design, develop, and optimize PySpark-based ETL pipelines on on-prem and cloud clusters.
• Build high-volume data ingestion frameworks using Kafka for real-time data.
• Tune and manage Hadoop ecosystem components (HDFS, YARN, MapReduce/Tez, Oozie/Airflow).
• Create high-performance Hive models for regulatory reporting and risk processing.
• Collaborate with quants, risk tech, and product owners; mentor junior engineers.

Skills

PySpark

Apache Kafka

Hadoop Ecosystem

Hive

Databricks Lakehouse

Delta Lake

SQL

Real-time Data Ingestion

Data Governance

CI/CD

Git

Jenkins

Bitbucket

Workflow Orchestration

Cloud & On-Prem Big Data

Education

Bachelor of Computer Science

Tools

Databricks

Hadoop

HDFS

YARN

MapReduce

Tez

Oozie

Airflow

Job description

Tata Consultancy Services in the United States is seeking a Senior Big Data Engineer with 10-13 years of hands-on experience, specializing in Hadoop, PySpark, Kafka, Hive, and Databricks Lakehouse. The role focuses on building scalable, well-governed data platforms for regulatory, risk, trading, and analytics workloads, with Delta Lake, Bronze/Silver/Gold modeling, and real-time ingestion.

On-prem and cloud deployments, collaboration with quants and risk teams, and leadership of junior engineers