Senior Data Platform & Product Engineering Lead

Tata Consultancy Services; Irvine (CA); On-site; USD 100,000 - 120,000; Full time

Job summary

Tata Consultancy Services in Irvine, CA seeks a Senior Data Platform / Data Product Engineering Lead to drive enterprise data product lifecycle enablement across modern data platforms. The role requires deep expertise in Databricks, Airflow, CI/CD automation, data governance, and data marketplace constructs.

You will lead platform transformation initiatives, mentor engineering teams, define self‑service data platform strategies, enable automated provisioning, and ensure secure, governed data

Qualifications

• 10+ years of experience in Data Engineering / Data Platform roles.
• Strong hands‑on expertise in Databricks (Delta Lake, workflows, DAG), Apache Airflow / Astronomer, Python, SQL, DBT, AWS.
• Proven experience implementing CI/CD frameworks (Harness, GitHub Actions, Azure DevOps).
• Deep understanding of data governance (catalogs, lineage, contracts, metadata), data quality and masking techniques, enterprise data platforms, marketplace ecosystems, API‑based integrations (e.g., entitlement systems like AccessCentral).
• Monitoring/observability tools experience (e.g., Datadog).

Responsibilities

• Platform Strategy & Self‑Service Enablement: Define and implement a self‑service data platform strategy to reduce onboarding friction; lead automated provisioning of Databricks workspaces, Airflow/Astronomer environments, and access entitlements; establish isolated, stable development environments; drive platform observability by integrating metrics into tools like Datadog.
• Data Discovery, Access & Governance: Architect and implement enterprise‑wide data discovery and marketplace enablement; drive adoption of data contracts, metadata standards, domain‑aligned catalogs (Unity Catalog); enable secure access to curated, masked datasets; implement tagging, access patterns, and entitlement automation; partner with risk/compliance teams to enforce regulatory and governance controls.
• Data Engineering, Curation & Orchestration: Lead design of scalable data ingestion, curation, and transformation frameworks; build and standardize modular, reusable frameworks such as LaunchLake templates, Airflow DAG libraries, and DBT‑based transformation models; ensure data quality, consistency, embedded governance, and concurrent development using standardized patterns.
• CI/CD, Automation & Deployment: Define and enforce standard CI/CD pipelines across data products using Harness (or equivalent) and Databricks Asset Bundles; automate DAG deployments, DBT pipeline releases; reduce manual interventions and improve release reliability with feedback loops, notifications, and monitoring.
• Data Product Publishing & Marketplace Enablement: Drive publishing of data products to Unity Catalog and Enterprise Data Marketplace; define and enforce documentation standards, data ownership models, versioning and contract management; enable cross‑domain data sharing with embedded governance and access controls.
• Operations, Observability & Reliability: Establish a scalable operating model for data product support; implement monitoring dashboards, data quality frameworks, usage and performance metrics; improve visibility into pipeline health, data lineage, access and consumption patterns; lead incident management, root‑cause analysis, and escalation processes.
• Transformation, Roadmap & Innovation: Drive execution of platform priorities such as data contract activation strategy, domain catalog integration, data masking in development environments, data quality frameworks, DBT adoption and proofs of concept; lead maturity uplift from manual, fragmented workflows to standardized, automated paved paths; champion continuous improvement and innovation in developer experience.

Skills

Databricks

Airflow

Python

SQL

DBT

AWS

CI/CD automation

Governance & data quality

Datadog

Tools

Astronomer

Harness

GitHub Actions

Azure DevOps

Job description

Tata Consultancy Services in Irvine, CA seeks a Senior Data Platform / Data Product Engineering Lead to drive enterprise data product lifecycle enablement across modern data platforms. The role requires deep expertise in Databricks, Airflow, CI/CD automation, data governance, and data marketplace constructs.

You will lead platform transformation initiatives, mentor engineering teams, define self‑service data platform strategies, enable automated provisioning, and ensure secure, governed data