Data Platform Tech Lead: Scale Pipelines & Lead Engineering

Tata Consultancy Services; Irvine (CA); On-site; USD 130,000 - 200,000; Full time

Job summary

Tata Consultancy Services in Irvine, CA seeks a Technical Lead – Data Engineering to guide an engineering team in building and scaling a robust data platform. You will stay hands-on with Databricks, dbt, and Airflow while mentoring engineers and conducting rigorous code reviews to deliver high-quality data products.

Lead data pipelines with PySpark, Delta Live Tables, and Spark SQL, enforcing standards and governance.

Qualifications

• 10+ years of professional experience in data engineering and backend software development.
• 2+ years in a technical lead, team lead, or mentoring capacity.

Responsibilities

• Lead a team of data engineers to execute sprint goals, managing code quality and delivery timelines.
• Act as SME for Databricks, dbt, and Airflow best practices.
• Enforce engineering standards, including code modularity, documentation, and version control.
• Conduct comprehensive code reviews for scalability, security, and performance.
• Build and maintain production-grade data pipelines using PySpark, Delta Live Tables, and Spark SQL.
• Develop complex dbt models, macros, and tests to transform raw data to analytics-ready layers.
• Author and schedule Apache Airflow DAGs with dynamic task mapping and custom operators.
• Optimize pipeline performance by troubleshooting Spark jobs, Delta table optimization, and Airflow scheduling.

Skills

Leadership

Mentoring

Code reviews

Communication

Tools

Databricks

dbt

Airflow

PySpark

Delta Live Tables

Spark SQL

Unity Catalog

Delta Lake

GitHub Actions

Job description

Tata Consultancy Services in Irvine, CA seeks a Technical Lead – Data Engineering to guide an engineering team in building and scaling a robust data platform. You will stay hands-on with Databricks, dbt, and Airflow while mentoring engineers and conducting rigorous code reviews to deliver high-quality data products.

Lead data pipelines with PySpark, Delta Live Tables, and Spark SQL, enforcing standards and governance.