DFS Tower Lead

Tata Consultancy Services; Irvine (CA); On-site; USD 110,000 - 140,000; Full time

Job summary

Tata Consultancy Services in Irvine, CA seeks an experienced DFS Tower Lead to provide L3 production support for mission-critical data platforms and pipelines. You will monitor S3, Athena, CloudWatch, Databricks, and Dremio while leading incident management, RCA, and continuous service improvement.

The role focuses on running complex data workloads, performing RCA, upgrading runtimes, and partnering with Data Engineering to ensure platform stability and data quality across environments.

Qualifications

• Extensive experience in L3 production support for enterprise data platforms.
• Strong incident management and RCA capabilities.
• Expertise with AWS-based data pipelines and Databricks.

Responsibilities

• Provide L3 production support for enterprise data platforms.
• Monitor AWS S3, Athena, CloudWatch; Databricks and Dremio environments.
• Troubleshoot complex production incidents and meet SLAs.
• Perform RCA and implement permanent corrective actions.
• Optimize ETL/ELT workflows with Airflow.
• Maintain Databricks clusters and runtimes.
• Develop SQL and Python scripts for investigation.
• Monitor health and performance; ensure data reliability.

Skills

AWS

S3

Athena

CloudWatch

Airflow

Databricks

Dremio

Python

SQL

GitHub

metadata

profiling

schema

Data quality checks

regression testing

Runtime upgrades

RCA fixes

Job description

Job Description

DFS Tower Lead

Must Have Technical/Functional Skills

AWS, S3, Athena, CloudWatch, Airflow, Databricks, Dremio, Python, SQL, GitHub, metadata, profiling, schema, data quality checks, regression testing, runtime upgrades and RCA fixes

Roles & Responsibilities

We are seeking an experienced L3 Production Support Engineer to provide advanced support for enterprise data platforms and critical data pipelines. The role requires expertise in AWS, Databricks, Airflow, Dremio, Python, SQL, and data quality frameworks, with a strong focus on incident management, RCA, platform stability, runtime upgrades, and continuous service improvement.

Key Responsibilities

• Provide L3 production support for enterprise data platforms and ensure high availability of business-critical data pipelines.
• Monitor and support data workloads running on AWS services (S3, Athena, CloudWatch), Databricks, and Dremio environments.
• Troubleshoot and resolve complex production incidents, ensuring compliance with defined SLAs and minimizing business impact.
• Perform detailed Root Cause Analysis (RCA) for recurring issues and implement permanent corrective actions.
• Analyze, debug, and optimize ETL/ELT workflows orchestrated through Apache Airflow.
• Support and maintain Databricks clusters, jobs, notebooks, workflows, and runtime configurations.
• Develop and execute SQL and Python scripts for issue investigation, data validation, and operational support activities.
• Monitor infrastructure and application health using AWS CloudWatch, identifying performance bottlenecks and system anomalies.
• Manage and support data residing in AWS S3 and queried through Athena, ensuring reliability and performance.
• Perform data profiling and metadata validation to ensure alignment with business and technical requirements.
• Implement and monitor data quality checks, reconciliation processes, and validation frameworks to maintain data integrity.
• Support schema changes, column additions, and data model enhancements while ensuring downstream application compatibility.
• Execute and validate regression testing during code deployments, platform changes, and infrastructure upgrades.
• Manage Databricks runtime upgrades, platform patching, and environment maintenance activities with minimal downtime.
• Utilize GitHub for source code management, version control, release tracking, and deployment support.
• Work closely with Data Engineering, Architecture, Business, and Infrastructure teams to resolve production issues and improve platform stability.
• Participate in on-call rotations and provide support for critical incidents during off-hours and planned maintenance windows.
• Develop operational runbooks, knowledge art icles, and support documentation to improve support efficiency.

Salary Range- $110,000-$140,000 a year