Senior Big Data Engineer - Hadoop, Hive, PySpark Databricks

Tata Consultancy Services; Charlotte (NC); On-site; USD 110,000 - 125,000; Full time

Benefits offered by this job

Comprehensive Medical Coverage

401K Plan

Training Reimbursement

Vacation and Holidays

Job summary

Tata Consultancy Services in Charlotte, NC seeks a Senior Big Data Engineer with 9-14 years of experience to architect and deliver scalable data platforms for large financial systems. You will design end-to-end data pipelines, enforce governance, and collaborate with quants, risk teams, and product owners across cloud and on-prem environments.

Key focus areas include Databricks Lakehouse, Delta Lake, Bronze/Silver/Gold modeling, and real-time ingestion using Kafka.

Qualifications

• 9-14 years of hands-on Big Data engineering experience.
• Strong expertise in Databricks Lakehouse and Delta Lake optimizations.
• Experience designing data solutions for large-scale financial systems.
• Proficiency with SQL on massive TB/PB datasets and CI/CD practices.

Responsibilities

• Design, develop, and optimize PySpark ETL pipelines on on-prem and cloud.
• Build high-volume ingestion frameworks using Kafka for real-time data.
• Develop and tune Hadoop ecosystem components (HDFS, YARN, MapReduce, Tez, Oozie/Airflow).
• Create high-performance Hive data models for regulatory and risk processing.

Skills

PySpark

Kafka

Hadoop

Hive

SQL

Databricks Lakehouse

Delta Lake

Data Modeling

ETL Pipelines

Real-time Ingestion

CI/CD

Cloud & On-Prem

Education

Bachelor of Computer Science

Tools

Git

Jenkins

Bitbucket

Airflow/Oozie

Unity Catalog

Job description

Tata Consultancy Services in Charlotte, NC seeks a Senior Big Data Engineer with 9-14 years of experience to architect and deliver scalable data platforms for large financial systems. You will design end-to-end data pipelines, enforce governance, and collaborate with quants, risk teams, and product owners across cloud and on-prem environments.

Key focus areas include Databricks Lakehouse, Delta Lake, Bronze/Silver/Gold modeling, and real-time ingestion using Kafka.