Senior Big Data Architect – Databricks Lakehouse & Kafka

Tata Consultancy Services; Plano (TX); On-site; USD 110,000 - 130,000; Full time

Benefits offered by this job

Annual incentive

Medical coverage

Parental leaves

Insurance options

Training reimbursement

Time off

Legal assistance

401K plan

Job summary

Tata Consultancy Services in Plano, TX is seeking a Senior Big Data Engineer with 10–13 years of hands-on experience to design and optimize large-scale data solutions for financial services.

The role requires expert PySpark, Kafka, Hadoop ecosystem, Hive, and Databricks Lakehouse architecture, including Bronze/Silver/Gold modeling, Delta Lake, and scalable pipelines. You will collaborate with quants, risk teams, and architects to deliver governed data platforms.

Qualifications

• 10–13 years of hands-on Big Data engineering experience.
• Expert skills in PySpark, Kafka, Hadoop ecosystem, Hive, and Databricks Lakehouse.
• Strong SQL knowledge on TB/PB-scale datasets and data modeling.

Responsibilities

• Design, develop, and optimize PySpark ETL pipelines on on-prem Hadoop clusters and cloud environments.
• Build high-volume ingestion frameworks using Kafka for real-time data.
• Tune Hadoop ecosystem components (HDFS, YARN, MapReduce/Tez, Oozie/Airflow).
• Build Hive data models for regulatory reporting and risk processing.
• Architect Bronze/Silver/Gold layer modeling in Databricks Lakehouse.

Skills

PySpark

Apache Kafka

Hadoop ecosystem

Hive

Databricks Lakehouse

Delta Lake

Big Data ETL

SQL

Git

Jenkins

Bitbucket

Real-time ingestion

Data governance

Cloud & on-prem platforms

Education

Bachelor of Computer Science

Tools

Oozie/Airflow

YARN

HDFS

MapReduce/Tez

Job description

Tata Consultancy Services in Plano, TX is seeking a Senior Big Data Engineer with 10–13 years of hands-on experience to design and optimize large-scale data solutions for financial services.

The role requires expert PySpark, Kafka, Hadoop ecosystem, Hive, and Databricks Lakehouse architecture, including Bronze/Silver/Gold modeling, Delta Lake, and scalable pipelines. You will collaborate with quants, risk teams, and architects to deliver governed data platforms.