Lead Data Engineer - IM

Tata Consultancy Services; Irvine (CA); On-site; USD 110,000 - 140,000; Full time

Job summary

Tata Consultancy Services in Irvine, CA is seeking a Lead Data Engineer for Information Management, with hands-on Databricks, dbt, and Python, to lead enterprise-scale data modernization initiatives.

You will design, build, and optimize scalable data pipelines, migrate legacy data platforms to a Databricks Lakehouse, implement CI/CD for data workflows, and mentor a team of engineers to deliver trusted datasets.

Qualifications

• 8–15 years of data engineering experience.
• Experience leading data platform migration and modernization programs.
• Hands-on experience with Databricks, dbt, and Python in large-scale environments.

Responsibilities

• Design, build, and maintain scalable data pipelines using Databricks, PySpark, SQL, and Python.
• Lead migration of legacy data platforms to Databricks Lakehouse and modern ELT patterns with dbt.
• Develop data quality, monitoring, and CI/CD for data workflows; mentor engineers.

Skills

Databricks

dbt

Python

PySpark

SQL

Airflow

Tools

Databricks Lakehouse

Delta Lake

Unity Catalog

Databricks Workflows

Structured Streaming

Medallion Architecture

Job description

Job Description

Lead Data Engineer - IM

Must Have Technical/Functional Skills

Data Bricks , EBT ,Airflow , Asset Management exp

Roles & Responsibilities

Lead Data Engineer – Information Management (IM)

Experience

8–15 Years

Role Overview

We are looking for a highly skilled Lead Data Engineer with strong hands‑on experience in Databricks, dbt, and Python, and a proven track record of leading and executing data platform migration and modernization programs. The ideal candidate will have experience migrating legacy data warehouses, ETL platforms, and cloud/on‑prem data ecosystems to a modern Databricks Lakehouse architecture using dbt‑based transformation frameworks. This is a hands‑on leadership role requiring deep technical expertise, solution design capabilities, and the ability to mentor engineering teams while driving enterprise‑scale data modernization initiatives.

Key Responsibilities

Data Engineering & Development

• Design, build, and maintain scalable data pipelines using Databricks, PySpark, SQL, and Python.
• Develop and optimize ELT/ETL processes for large‑scale data ingestion and transformation.
• Implement robust data quality, monitoring, and reconciliation frameworks.
• Build reusable data engineering components and frameworks.

Data Platform Migration & Modernization

• Lead migration of legacy data platforms, data warehouses, and ETL ecosystems to Databricks Lakehouse.
• Transform existing ETL workloads to modern ELT patterns using dbt.
• Analyze source environments and define migration approaches, roadmap, and execution strategy.
• Drive code conversion, performance optimization, and workload modernization efforts.

Databricks Engineering

Develop Solutions Leveraging

• Databricks Lakehouse Platform
• Delta Lake
• Unity Catalog
• Databricks Workflows
• Structured Streaming
• Medallion Architecture (Bronze, Silver, Gold)

Optimize Spark jobs and Databricks workloads for performance and cost efficiency.

dbt Development

• Build and maintain dbt models, macros, tests, and documentation.
• Implement incremental processing and reusable transformation frameworks.
• Establish data lineage, testing, and CI/CD best practices.
• Work closely with business and analytics teams to build trusted datasets.

Leadership & Collaboration

• Lead a team of data engineers and developers.
• Perform code reviews and enforce engineering standards.
• Collaborate with arch itects, product owners, business analysts, and stakeholders.
• Mentor junior engineers and drive adoption of best practices.

Salary Range- $110,000-$140,000 a year