Technical Lead-Data Engineer

Tata Consultancy Services; Irvine (CA); On-site; USD 130,000 - 200,000; Full time

Job summary

Tata Consultancy Services in Irvine, CA seeks a Technical Lead – Data Engineering to guide an engineering team in building and scaling a robust data platform. You will stay hands-on with Databricks, dbt, and Airflow while mentoring engineers and conducting rigorous code reviews to deliver high-quality data products.

Lead data pipelines with PySpark, Delta Live Tables, and Spark SQL, enforcing standards and governance.

Qualifications

• 10+ years of professional experience in data engineering and backend software development.
• 2+ years in a technical lead, team lead, or mentoring capacity.

Responsibilities

• Lead a team of data engineers to execute sprint goals, managing code quality and delivery timelines.
• Act as SME for Databricks, dbt, and Airflow best practices.
• Enforce engineering standards, including code modularity, documentation, and version control.
• Conduct comprehensive code reviews for scalability, security, and performance.
• Build and maintain production-grade data pipelines using PySpark, Delta Live Tables, and Spark SQL.
• Develop complex dbt models, macros, and tests to transform raw data to analytics-ready layers.
• Author and schedule Apache Airflow DAGs with dynamic task mapping and custom operators.
• Optimize pipeline performance by troubleshooting Spark jobs, Delta table optimization, and Airflow scheduling.

Skills

Leadership

Mentoring

Code reviews

Communication

Tools

Databricks

dbt

Airflow

PySpark

Delta Live Tables

Spark SQL

Unity Catalog

Delta Lake

GitHub Actions

Job description

Job Description

Technical Lead – Data Engineering

Role Overview

We are looking for a highly skilled Tech Lead – Data Engineering to guide our engineering team in building and scaling a robust, modern data platform. In this role, you will be the bridge between architectural blueprints and engineering execution. You will remain deeply hands‑on with our core stack—Databricks, dbt, and Apache Airflow—while mentoring engineers, conducting rigorous code reviews, and ensuring the delivery of high-quality data products.

Key Responsibilities

Technical Leadership & Delivery

• Lead a team of data engineers to execute sprint goals, managing code quality and delivery timelines.
• Act as the subject matter expert (SME) for the team on Databricks, dbt, and Airflow best practices.
• Enforce engineering standards, including code modularity, documentation, and version control.
• Conduct comprehensive code reviews to ensure scalability, security, and performance.

Hands‑on Engineering & Optimization

• Build and maintain production‑grade data pipelines using PySpark, Delta Live Tables (DLT), and Spark SQL on Databricks.
• Develop complex dbt models, custom macros, and tests to transform raw data into analytics‑ready layers.
• Author and schedule sophisticated Apache Airflow DAGs, utilizing dynamic task mapping and custom operators.
• Optimize pipeline performance by troubleshooting bottlenecked Spark jobs, Z‑Ordering Delta tables, and tuning Airflow schedulers.

DataOps & Governance

• Implement and manage CI/CD pipelines for data deployments using toolsets like GitHub Actions or Azure DevOps.
• Enforce data governance policies, access controls, and lineage tracking via Databricks Unity Catalog.
• Integrate automated data quality testing and alerting mechanisms across dbt and Airflow workflows.

Mentorship & Collaboration

• Mentor and upskill junior and mid‑level data engineers through pair programming and workshops.
• Translate architectural designs into actionable, granular engineering tasks and JIRA tickets.
• Collaborate closely with data architects, product managers, and downstream data analysts.

Required Qualifications

• 10+ years of professional experience in data engineering and backend software development.
• 2+ years of experience in a technical lead, team lead, or mentoring capacity.

Technical Proficiencies

• Databricks: Strong hands‑on experience with Delta Lake, Unity Catalog, and optimizing PySpark workloads.
• dbt: Proficiency with dbt Core or Cloud, including advanced macros, packages, and custom testing.
• Airflow: Solid experience writing complex, rel iable DAGs, handling task failures, and managing dependencies.
• Languages: Advanced proficiency in Python and expert‑level SQL writing.
• Cloud Platforms: Hands‑on experience deploying data solutions on at least one cloud provider (AWS, Azure, or GCP).

Leadership Skills

• Proven ability to guide engineering sprint velocity and resolve technical blockers for a team.
• Excellent verbal and written communication skills to articulate technical trade‑offs.

Salary Range- $130,000-$200,000 a year