Dfs Lead

Tata Consultancy Services; Irvine (CA); Hybrid; USD 110.000 - 140.000; Vollzeit

Zusammenfassung

Tata Consultancy Services Limited in Irvine, CA is seeking a Data Factory Services Lead to own end‑to‑end enterprise data pipelines and production readiness. You will stay hands‑on in Python and SQL, review designs, and guide onshore/offshore teams to ensure secure, scalable solutions across AWS, Airflow, Databricks, and Dremio.

This senior role combines architectural leadership with hands‑on delivery, incident response, and mentorship, operating in a hybrid Irvine/San Antonio‑style model with

Qualifikationen

• Minimum 10 years of relevant technology experience.
• Proven production experience across AWS, Airflow, Pepperdata, Databricks, Dremio, Python, SQL, GitHub, and VS Code.
• Willingness to work from Irvine office Tue–Thu under a hybrid model.

Aufgaben

• Lead architecture, design, development, deployment, and production support of enterprise data factory services and data pipelines.
• Remain hands‑on in Python and SQL development, code review, debugging, performance tuning, automation, and incident resolution.
• Design and support AWS solutions using S3, CodeBuild, Athena, CloudWatch, Secrets Manager.
• Build, schedule, monitor, and troubleshoot Apache Airflow workflows, including DAG design and retries.
• Use Pepperdata for workload visibility and optimization of distributed data workloads.
• Develop Databricks workloads for scalable data engineering and analytics.
• Use Dremio for governed, high‑performance data access and semantic design.
• Create efficient SQL for ingestion, transformation, analytics, and reporting.
• Develop CI/CD automation through GitHub and CodeBuild; manage branches and releases.
• Use VS Code as primary IDE for Python/SQL, config, testing, and debugging.

Kenntnisse

AWS Cloud

Workflow Orchestration

Production Engineering

Performance Optimization

Data & Analytics Platforms

Programming & Query Engineering

Toolchain

Tools

Apache Airflow

Pepperdata

Databricks

Dremio

GitHub

Visual Studio Code

Jobbeschreibung

Data Factory Services Lead

Role Summary

The Data Factory Services Lead is a senior, hands‑on onshore technical leader responsible for designing, engineering, deploying, supporting, and continuously improving enterprise data platforms and pipelines. The role owns the complete technical flow across AWS services, workflow orchestration, distributed data processing, query and analytics platforms, CI/CD, observability, and production support. The lead will provide technical direction to distributed teams, conduct design and code reviews, resolve complex production issues, and ensure solutions are secure, scalable, reliable, maintainable, and operationally ready.

Key Responsibilities

• Lead architecture, design, development, enhancement, deployment, and production support of enterprise data factory services and data pipelines.
• Remain actively hands‑on in Python and SQL development, code review, debugging, performance tuning, automation, and incident resolution.
• Design and support AWS solutions using Amazon S3, AWS CodeBuild, Amazon Athena, Amazon CloudWatch, and AWS Secrets Manager.
• Build, schedule, monitor, and troubleshoot Apache Airflow workflows, including DAG design, dependencies, retries, alerting, backfills, recovery, and operational controls.
• Use Pepperdata for workload visibility, performance analysis, capacity insights, and optimization of distributed data workloads.
• Design, develop, optimize, and support Databricks workloads for scalable data engineering, transformation, and analytics use cases.
• Use Dremio to enable governed, high-performance access to enterprise data, including semantic-layer design, query optimization, and issue resolution.
• Develop efficient SQL for ingestion, transformation, reconciliation, analytics, validation, and operational reporting across large datasets.
• Create and maintain CI/CD automation through GitHub and AWS CodeBuild, including source control, branching, pull requests, build validation, release controls, and rollback readiness.
• Use Visual Studio Code as a primary engineering environment for Python, SQL, configuration, testing, debugging, and repository-based development.
• Implement monitoring, logging, alerting, and operational dashboards using Amazon CloudWatch and platform‑native capabilities.
• Act as the senior escalation point for data, workflow, cloud, query, performance, deployment, and production incidents; lead root-cause analysis and permanent corrective actions.
• Guiding offshore engineers through technical design, coding standards, peer reviews, troubleshooting, deployment readiness, documentation, and knowledge transfer.
• Partner with application, data, cloud, DevOps, security, testing, support, architecture, and business stakeholders to deliver dependable services.
• Maintain architecture notes, runbooks, SOPs, dependency maps, production support procedures, release records, and operational knowledge artifacts.

Required Technical Skills - All Mandatory

• AWS Cloud: Amazon S3, AWS CodeBuild, Amazon Athena, Amazon CloudWatch, and AWS Secrets Manager. Candidate must be able to design, configure, troubleshoot, monitor, secure, and support these services in production.
• Workflow Orchestration: Apache Airflow: DAG engineering, scheduling, dependencies, retries, backfills, monitoring, alerting, failure recovery, and production troubleshooting.
• Production Engineering: Observability, secure secrets handling, incident management, root-cause analysis, performance tuning, release validation, reliability, and operational readiness.
• Performance Optimization: Pepperdata: workload monitoring, performance analysis, capacity insights, troubleshooting, and optimization of distributed data workloads.
• Data & Analytics Platforms: Databricks and Dremio: hands‑on data engineering, integrations, query performance, semantic access, platform troubleshooting, and production support.
• Programming & Query Engineering: Python and SQL: advanced development, automation, testing, debugging, optimization, data validation, and scalable processing.
• Toolchain: GitHub and Visual Studio Code: repository management, branching, pull requests, reviews, version control, debugging, and collaborative development.

Leadership and Collaboration Skills

• Demonstrated ability to lead through hands‑on technical contribution, sound engineering judgment, and end‑to‑end accountability.
• Experience leading distributed onshore and offshore delivery teams and translating business and technical requirements into executable plans.
• Ability to conduct architecture, code, data pipeline, SQL, security, deployment, and operational‑readiness reviews.
• Strong stakeholder communication, analytical problem‑solving, decision‑making, prioritization, and escalation‑management skills.
• Ability to coordinate critical production incidents, planned implementations, post‑release validation, and service stabilization.
• Ability to mentor engineers, enforce engineering standards, remove technical blockers, and build sustainable operational knowledge.

Required Qualifications

• Minimum 10 years of relevant technology experience, with recent experience in a senior technical lead or comparable hands‑on leadership role.
• Demonstrable production experience across all required technologies : AWS S3, CodeBuild, Athena, CloudWatch, Secrets Manager, Airflow, Pepperdata, Databricks, Dremio, Python, SQL, GitHub, and Visual Studio Code.
• Experience owning enterprise data services through design, build, release, monitoring, support, remediation, and continuous improvement.
• Willingness and ability to work from the San Antonio or Irvine office Tuesday through Thursday under the stated hybrid model.

Preferred Experience

• Experience delivering enterprise data platforms in regulated or audit‑sensitive environments.
• Knowledge of data governance, access control, secure engineering, platform reliability, release governance, and operational controls.
• Experience supporting high‑volume or business‑critical data services with formal incident, problem, and change‑management practices.

Salary Range- $110,000-$140,000 a year

Location

Irvine, CA

Function

TECHNOLOGY

Role

Technical Lead

ID

437486

Skills

AWS

Salary Range

$110,000 - $140,000