Senior Data Engineer - Databricks Lakehouse Expert

Tata Consultancy Services Limited; Irvine (CA); On-site; USD 120,000 - 180,000; Full time

Job summary

Tata Consultancy Services Limited in Irvine, CA is seeking a hands-on Senior Data Engineer to design, build, and optimize enterprise-scale data solutions on the Databricks Lakehouse Platform. The role focuses on batch and streaming pipelines, modernizing legacy ETL, and delivering governed data products for analytics, reporting, and risk management.

The ideal candidate will have deep experience with Databricks, Spark, PySpark, SQL, Python, dbt, Apache Airflow, Delta Lake, Unity Catalog, and

Qualifications

• Typically 7–10 years of data engineering or distributed data-processing experience.
• Strong hands-on experience with Databricks, Spark, PySpark, Delta Lake, Python, and advanced SQL.
• Experience building production-grade ETL/ELT pipelines for large datasets.
• Experience with dbt Core or dbt Cloud including models, macros, tests, and incremental processing.
• Experience with orchestration platforms like Apache Airflow or Databricks Workflows.
• Experience with Unity Catalog, data lineage, access controls, and data governance.

Responsibilities

• Design, build, test, deploy, and maintain scalable ETL and ELT pipelines using Databricks, PySpark, Spark SQL, Python, and SQL.
• Develop reusable ingestion and transformation frameworks for structured, semi-structured, and streaming data.
• Implement batch, incremental, change-data-capture, and streaming processing patterns.
• Build and maintain Delta Lake tables using medallion architecture across Bronze, Silver, and Gold layers.
• Develop dbt models, tests, macros, packages, documentation, and incremental processing patterns.
• Create and maintain Apache Airflow DAGs and Databricks Workflows with dependency management, retries, alerting, and operational controls.
• Integrate data from APIs, databases, files, event streams, and cloud data services.
• Produce technical designs, mapping specifications, lineage documentation, deployment instructions, and operational runbooks.

Skills

Databricks

Apache Spark

PySpark

Python

SQL

dbt

Airflow

Delta Lake

Unity Catalog

Cloud data services

Tools

dbt Core/Cloud

Git

Terraform

Databricks Workflows

Apache Airflow

Job description

Tata Consultancy Services Limited in Irvine, CA is seeking a hands-on Senior Data Engineer to design, build, and optimize enterprise-scale data solutions on the Databricks Lakehouse Platform. The role focuses on batch and streaming pipelines, modernizing legacy ETL, and delivering governed data products for analytics, reporting, and risk management.

The ideal candidate will have deep experience with Databricks, Spark, PySpark, SQL, Python, dbt, Apache Airflow, Delta Lake, Unity Catalog, and