Data Platform Production Lead — AWS/Databricks & Airflow

Veriipro; Irvine (CA); On-site; USD 130,000 - 195,000; Full time

Job summary

Veriipro seeks an experienced DFS Tower Lead / L3 Production Support Engineer to provide technical leadership and advanced production support for enterprise data platforms and data pipelines. You will manage incidents, perform RCA, and drive platform stability across AWS, Databricks, Dremio, and Airflow.

The role emphasizes automation, data quality checks, and collaboration with cross-functional teams to ensure continuous service improvement.

Qualifications

• Strong experience with AWS, S3, Athena, and CloudWatch.
• Hands-on experience with Databricks, Dremio, and Apache Airflow.
• Strong programming and troubleshooting skills in Python and SQL.
• Experience with GitHub for source control and deployment support.
• Strong knowledge of metadata, data profiling, schema validation, and data quality checks.
• Experience with regression testing, runtime upgrades, patching, and environment maintenance.
• Strong expertise in production support, incident management, RCA, and permanent corrective actions.

Responsibilities

• Lead L3 production support for enterprise data platforms and business-critical data pipelines.
• Monitor and support workloads across AWS, Databricks, Dremio, and Airflow environments.
• Troubleshoot complex production incidents and ensure resolution within defined SLAs.
• Perform detailed Root Cause Analysis (RCA) and implement permanent fixes.
• Analyze, debug, and optimize Airflow ETL/ELT workflows.
• Support Databricks clusters, jobs, notebooks, workflows, and runtime configurations.
• Use Python and SQL for troubleshooting, data validation, analysis, and operational activities.
• Monitor platform health and performance using AWS CloudWatch.
• Support data stored in S3 and queried through Athena.
• Perform data profiling, metadata validation, schema validation, and data quality checks.
• Support schema changes, column additions, and data model enhancements while ensuring downstream compatibility.
• Execute regression testing for deployments, platform changes, and infrastructure upgrades.
• Lead/support Databricks runtime upgrades, patching, and environment maintenance.
• Use GitHub for version control, release tracking, and deployment support.
• Collaborate with Data Engineering, Architecture, Business, and Infrastructure teams to resolve issues and improve platform stability.
• Participate in on-call support and critical incident management as required.
• Create and maintain operational runbooks, knowledge articles, and support documentation.
• Drive continuous improvement, automation, operational efficiency, and service stability across the data platform.

Skills

Python

SQL

Data quality

Root Cause Analysis

Incident management

Leadership

GitHub

Tools

Databricks

Dremio

Apache Airflow

AWS

S3

Athena

CloudWatch

GitHub

Job description

Veriipro seeks an experienced DFS Tower Lead / L3 Production Support Engineer to provide technical leadership and advanced production support for enterprise data platforms and data pipelines. You will manage incidents, perform RCA, and drive platform stability across AWS, Databricks, Dremio, and Airflow.

The role emphasizes automation, data quality checks, and collaboration with cross-functional teams to ensure continuous service improvement.