Technical Lead

Tata Consultancy Services; Plano (TX); On-site; USD 110,000 - 125,000; Full time

Benefits offered by this job

Discretionary Annual Incentive

Comprehensive Medical Coverage

Family Support: Maternal & Parental Le

Insurance Options: Auto & Home

Certification & Training Reimbursement

Time Off: Vacation & Holidays

Legal & Financial Assistance

401K Plan

Job summary

Tata Consultancy Services in the United States is seeking a Senior Big Data Engineer with 10-13 years of hands-on experience, specializing in Hadoop, PySpark, Kafka, Hive, and Databricks Lakehouse. The role focuses on building scalable, well-governed data platforms for regulatory, risk, trading, and analytics workloads, with Delta Lake, Bronze/Silver/Gold modeling, and real-time ingestion.

On-prem and cloud deployments, collaboration with quants and risk teams, and leadership of junior engineers

Qualifications

• 10-13 years of hands-on experience in Big Data engineering.
• Expertise in PySpark, Kafka, Hive, and Databricks Lakehouse architectures.
• Strong knowledge of Delta Lake, data modeling, and governance practices.
• Experience with on-prem and cloud big data platforms and CI/CD pipelines.

Responsibilities

• Design, develop, and optimize PySpark-based ETL pipelines on on-prem and cloud clusters.
• Build high-volume data ingestion frameworks using Kafka for real-time data.
• Tune and manage Hadoop ecosystem components (HDFS, YARN, MapReduce/Tez, Oozie/Airflow).
• Create high-performance Hive models for regulatory reporting and risk processing.
• Collaborate with quants, risk tech, and product owners; mentor junior engineers.

Skills

PySpark

Apache Kafka

Hadoop Ecosystem

Hive

Databricks Lakehouse

Delta Lake

SQL

Real-time Data Ingestion

Data Governance

CI/CD

Git

Jenkins

Bitbucket

Workflow Orchestration

Cloud & On-Prem Big Data

Education

Bachelor of Computer Science

Tools

Databricks

Hadoop

HDFS

YARN

MapReduce

Tez

Oozie

Airflow

Job description

• Delta Change Data Feed (CDF) o schema evolution & enforcement o ACID transaction handling

Job Description

Must Have Technical/Functional Skills

Primary skills: PySpark, Apache Kafka, Hadoop Ecosystem, Hive, Databricks Lakehouse Architecture, Delta Lake, Bronze/Silver/Gold Data Modeling, Big Data ETL Pipeline Development, SQL, Real-time Data Ingestion Frameworks, Data Governance & Cataloging, CI/CD Tools - Git, Jenkins, Bitbucket, Workflow Orchestration, and Cloud & On-Prem Big Data Platforms.

Experience: Minimum 10+ years

Roles & Responsibilities

Seeking a Senior Big Data Engineer with 10-13 years of experience specializing in Hadoop, PySpark, Kafka, Hive, and strong experience designing data solutions for large-scale financial systems.

In addition, the candidate must possess advanced expertise in Databricks Lakehouse architecture, particularly around Bronze/Silver/Gold layer data modeling, Delta Lake optimizations, and building reliable, scalable pipelines for regulatory, risk, trading, and analytics workloads.

This role focuses on delivering highly performant, well-governed data platforms that support the bank’s mission-critical global markets functions.

Key Responsibilities

Big Data Platform Engineering

• Design, develop, and optimize PySpark-based ETL pipelines running on on-prem Hadoop clusters and cloud environments.
• Build high-volume ingestion frameworks using Kafka for real-time and near-real-time trading and market data.
• Develop, tune, and manage Hadoop ecosystem components—HDFS, YARN, MapReduce, Tez, Oozie/Airflow.
• Build high-performance, optimized Hive data models for regulatory reporting, trade lifecycle, and market risk processing.

Databricks Lakehouse & Delta Framework

• Architect and implement Bronze/Silver/Gold layer modeling patterns within the Databricks Lakehouse.
• Apply Delta Lake best practices including:
• optimized file management
• Z-Ordering
• Delta Change Data Feed (CDF) o schema evolution & enforcement o ACID transaction handling
• Build reusable frameworks for ingestion, cleansing, transformation, and consumption of data across Lakehouse layers.
• Enable governance, lineage, and auditability using Unity Catalog or equivalent cataloging tools.

Collaboration, Leadership & Delivery

• Collaborate closely with quants, product owners, architects, risk tech, and business users.
• Participate in agile ceremonies - sprint planning, refinement, design reviews.
• Mentor junior engineers and contribute to building strong engineering practices across tech teams.

Required Skills & Experience

• 10-13 years of hands-on experience in Big Data engineering.
• Expert skills in:
• PySpark - dataframe optimizations, partitioning, broadcast strategies, distributed computing.
• Kafka - producer/consumer design, schema registry, streaming ETLs.
• Hadoop ecosystem - HDFS, YARN, MapReduce/Tez, Oozie/Airflow.
• Hive - advanced query tuning, TEZ optimization, partition/bucket management.
• Extensive hands-on experience with Databricks Lakehouse, including:
• Bronze/Silver/Gold layer modeling
• Delta Lake optimizations
• Data quality frameworks on Lakehouse
• Structured & unstructured data handling
• Experience in Global Markets, Risk, Treasury, Trade Surveillance, or Regulatory Reporting.
• Strong SQL knowledge with experience working on massive datasets (TB/PB scale).

Experience with CI/CD practices - Git, Jenkins, Bitbucket, build pipelines.

TCS Employee Benefits Summary

• Discretionary Annual Incentive.
• Comprehensive Medical Coverage: Medical & Health, Dental & Vision, Disability Planning & Insurance, Pet Insurance Plans.
• Family Support: Maternal & Parental Leaves.
• Insurance Options: Auto & Home Insurance, Identity Theft Protection.
• Convenience & Professional Growth: Commuter Benefits & Certification & Training Reimbursement.
• Time Off: Vacation, Time Off, Sick Leave & Holidays.
• Legal & Financial Assistance: Legal Assistance, 401K Plan, Performance Bonus, College Fund, Student Loan Refinancing.
• Salary Range: $110,000- 125,000 a year

Qualifications

BACHELOR OF COMPUTER SCIENCE