Senior Data Factory Lead — AWS/Data Pipelines Architect

Tata Consultancy Services Limited; Irvine (CA); Hybrid; USD 110,000 - 140,000; Full time

Job summary

Tata Consultancy Services Limited in Irvine, CA is seeking a Data Factory Services Lead to own end‑to‑end enterprise data pipelines and production readiness. You will stay hands‑on in Python and SQL, review designs, and guide onshore/offshore teams to ensure secure, scalable solutions across AWS, Airflow, Databricks, and Dremio.

This senior role combines architectural leadership with hands‑on delivery, incident response, and mentorship, operating in a hybrid Irvine/San Antonio‑style model with

Qualifications

• Minimum 10 years of relevant technology experience.
• Proven production experience across AWS, Airflow, Pepperdata, Databricks, Dremio, Python, SQL, GitHub, and VS Code.
• Willingness to work from Irvine office Tue–Thu under a hybrid model.

Responsibilities

• Lead architecture, design, development, deployment, and production support of enterprise data factory services and data pipelines.
• Remain hands‑on in Python and SQL development, code review, debugging, performance tuning, automation, and incident resolution.
• Design and support AWS solutions using S3, CodeBuild, Athena, CloudWatch, Secrets Manager.
• Build, schedule, monitor, and troubleshoot Apache Airflow workflows, including DAG design and retries.
• Use Pepperdata for workload visibility and optimization of distributed data workloads.
• Develop Databricks workloads for scalable data engineering and analytics.
• Use Dremio for governed, high‑performance data access and semantic design.
• Create efficient SQL for ingestion, transformation, analytics, and reporting.
• Develop CI/CD automation through GitHub and CodeBuild; manage branches and releases.
• Use VS Code as primary IDE for Python/SQL, config, testing, and debugging.

Skills

AWS Cloud

Workflow Orchestration

Production Engineering

Performance Optimization

Data & Analytics Platforms

Programming & Query Engineering

Toolchain

Tools

Apache Airflow

Pepperdata

Databricks

Dremio

GitHub

Visual Studio Code

Job description

Tata Consultancy Services Limited in Irvine, CA is seeking a Data Factory Services Lead to own end‑to‑end enterprise data pipelines and production readiness. You will stay hands‑on in Python and SQL, review designs, and guide onshore/offshore teams to ensure secure, scalable solutions across AWS, Airflow, Databricks, and Dremio.

This senior role combines architectural leadership with hands‑on delivery, incident response, and mentorship, operating in a hybrid Irvine/San Antonio‑style model with