Senior Big Data Engineer: Databricks Lakehouse & PySpark

Tata Consultancy Services; Plano (TX); On-site; USD 80,000 - 140,000; Full time

Benefits offered by this job

Discretionary Annual Incentive

Comprehensive Medical Coverage

Dental & Vision Coverage

401K Plan

Job summary

Tata Consultancy Services is seeking a Senior Big Data Engineer with 10–13 years of experience to design and implement scalable data platforms for global markets. The role emphasizes PySpark, Kafka, Hive, and Databricks Lakehouse architectures across on‑prem and cloud environments.

Responsibilities include building end‑to‑end data pipelines, ensuring governance and quality, and collaborating with quants and business users to deliver compliant financial data solutions.

Qualifications

• 10–13 years of hands-on Big Data engineering experience.
• Strong expertise in Databricks Lakehouse architecture and Delta Lake.
• Experience designing data solutions for large-scale financial systems.

Responsibilities

• Design, develop, and optimize PySpark-based ETL pipelines on on‑prem Hadoop clusters and cloud environments.
• Build high‑volume ingestion frameworks using Kafka for real-time data.
• Develop, tune, and manage Hadoop ecosystem components (HDFS, YARN, MapReduce/Tez, Oozie/Airflow).
• Build high-performance Hive data models for regulatory reporting, trade lifecycle, and market risk processing.

Skills

PySpark

Kafka

Hadoop Ecosystem

Hive

Databricks Lakehouse

Delta Lake

Bronze/Silver/Gold modeling

Big Data ETL pipelines

SQL

Real-time data ingestion

Data governance & cataloging

Cloud & on-prem platforms

Education

Bachelor's degree in Computer Science or related field

Tools

Git

Jenkins

Bitbucket

Airflow

Oozie

Job description

Tata Consultancy Services is seeking a Senior Big Data Engineer with 10–13 years of experience to design and implement scalable data platforms for global markets. The role emphasizes PySpark, Kafka, Hive, and Databricks Lakehouse architectures across on‑prem and cloud environments.

Responsibilities include building end‑to‑end data pipelines, ensuring governance and quality, and collaborating with quants and business users to deliver compliant financial data solutions.