ETL Developer

Tata Consultancy Services; Cleveland (OH); On-site; USD 80,000 - 140,000; Full time

Benefits offered by this job

Discretionary Annual Incentive

Medical Coverage

Family Leaves

Insurance Options

Certification & Training Reimbursement

Vacation & Holidays

Legal & Financial Assistance

Job summary

Tata Consultancy Services in Cleveland, OH is seeking a Senior Big Data Engineer with 10–13 years of experience to design and deliver scalable data platforms for large financial systems.

You will work with PySpark, Kafka, Hadoop, Hive, and Databricks Lakehouse, building Bronze/Silver/Gold data models, Delta Lake optimizations, and reliable pipelines.

This role emphasizes governance, performance, and collaboration with quants, risk teams, and product owners in an agile environment.

Qualifications

• Minimum 10+ years of experience in Big Data engineering.
• Expert in PySpark, Kafka, Hadoop ecosystem, and Databricks Lakehouse.
• Experience designing data solutions for large-scale financial systems.
• Strong SQL and data modeling skills in TB/PB scale.

Responsibilities

• Design, develop, and optimize PySpark-based ETL pipelines running on on-prem Hadoop clusters and cloud environments.
• Build high-volume ingestion frameworks using Kafka for real-time and near-real-time trading and market data.
• Develop, tune, and manage Hadoop ecosystem components - HDFS, YARN, MapReduce, Tez, Oozie/Airflow.
• Build high-performance, optimized Hive data models for regulatory reporting, trade lifecycle, and market risk processing.
• Architect and implement Bronze/Silver/Gold layer modeling patterns within the Databricks Lakehouse.

Skills

PySpark

Kafka

Hadoop Ecosystem

Hive

Databricks Lakehouse Architecture

Delta Lake

Bronze/Silver/Gold Data Modeling

Big Data ETL Pipeline Development

SQL

Real-time Data Ingestion Frameworks

Data Governance & Cataloging

CI/CD Tools - Git

Jenkins

Bitbucket

Workflow Orchestration

Cloud & On-Prem Big Data Platforms

Education

Bachelor of Computer Science

Job description

• Delta Change Data Feed (CDF) o schema evolution & enforcement o ACID transaction handling

Job Description

Must Have Technical/Functional Skills

Primary skills: PySpark, Apache Kafka, Hadoop Ecosystem, Hive, Databricks Lakehouse Architecture, Delta Lake, Bronze/Silver/Gold Data Modeling, Big Data ETL Pipeline Development, SQL, Real-time Data Ingestion Frameworks, Data Governance & Cataloging, CI/CD Tools - Git, Jenkins, Bitbucket, Workflow Orchestration, and Cloud & On-Prem Big Data Platforms.

Experience: Minimum 10+ years

Roles & Responsibilities

Seeking a Senior Big Data Engineer with 10-13 years of experience specializing in Hadoop, PySpark, Kafka, Hive, and strong experience designing data solutions for large-scale financial systems.

In addition, the candidate must possess advanced expertise in Databricks Lakehouse architecture, particularly around Bronze/Silver/Gold layer data modeling, Delta Lake optimizations, and building reliable, scalable pipelines for regulatory, risk, trading, and analytics workloads.

This role focuses on delivering highly performant, well-governed data platforms that support the bank’s mission-critical global markets functions.

Key Responsibilities

Big Data Platform Engineering

• Design, develop, and optimize PySpark-based ETL pipelines running on on-prem Hadoop clusters and cloud environments.
• Build high-volume ingestion frameworks using Kafka for real-time and near-real-time trading and market data.
• Develop, tune, and manage Hadoop ecosystem components - HDFS, YARN, MapReduce, Tez, Oozie/Airflow.
• Build high-performance, optimized Hive data models for regulatory reporting, trade lifecycle, and market risk processing.

Databricks Lakehouse & Delta Framework

• Architect and implement Bronze/Silver/Gold layer modeling patterns within the Databricks Lakehouse.
• Apply Delta Lake best practices including:
• optimized file management
• Z-Ordering
• Delta Change Data Feed (CDF) o schema evolution & enforcement o ACID transaction handling
• Build reusable frameworks for ingestion, cleansing, transformation, and consumption of data across Lakehouse layers.
• Enable governance, lineage, and auditability using Unity Catalog or equivalent cataloging tools.

Collaboration, Leadership & Delivery

• Collaborate closely with quants, product owners, architects, risk tech, and business users.
• Participate in agile ceremonies - sprint planning, refinement, design reviews.
• Mentor junior engineers and contribute to building strong engineering practices across tech teams.

Required Skills & Experience

• 10-13 years of hands-on experience in Big Data engineering.
• Expert skills in:
• PySpark - dataframe optimizations, partitioning, broadcast strategies, distributed computing.
• Kafka - producer/consumer design, schema registry, streaming ETLs.
• Hadoop ecosystem - HDFS, YARN, MapReduce/Tez, Oozie/Airflow.
• Hive - advanced query tuning, TEZ optimization, partition/bucket management.
• Extensive hands-on experience with Databricks Lakehouse, including:
• Bronze/Silver/Gold layer modeling
• Delta Lake optimizations
• Data quality frameworks on Lakehouse
• Structured & unstructured data handling
• Experience in Global Markets, Risk, Treasury, Trade Surveillance, or Regulatory Reporting.
• Strong SQL knowledge with experience working on massive datasets (TB/PB scale).

Experience with CI/CD practices - Git, Jenkins, Bitbucket, build pipelines.

TCS Employee Benefits Summary

• Discretionary Annual Incentive.
• Comprehensive Medical Coverage: Medical & Health, Dental & Vision, Disability Planning & Insurance, Pet Insurance Plans.
• Family Support: Maternal & Parental Leaves.
• Insurance Options: Auto & Home Insurance, Identity Theft Protection.
• Convenience & Professional Growth: Commuter Benefits & Certification & Training Reimbursement.
• Time Off: Vacation, Time Off, Sick Leave & Holidays.
• Legal & Financial Assistance: Legal Assistance, 401K Plan, Performance Bonus, College Fund, Student Loan Refinancing.

Salary Range: $80,000- 140,000 a year

Qualifications

BACHELOR OF COMPUTER SCIENCE