Senior Data QA Automation Engineer — Lakehouse

V4C.ai; United States; Remote; USD 140,000 - 190,000; Full time

Job summary

V4C.ai in the United States seeks a Senior Data QA Automation Engineer to lead the quality strategy for our Databricks Lakehouse platforms, ensuring data quality across Delta Lakes, ETL pipelines, and enterprise data models.

You will mentor engineers, design scalable test frameworks, and own end-to-end data validation, governance verification, and CI/CD integration within Azure DevOps or GitHub Actions. This role drives data quality culture and engineering excellence.

Qualifications

• 8+ years in data engineering, data QA, or SDET.
• 2+ years architecting test automation in Databricks.
• Proficient in PySpark, Python, and SQL.

Responsibilities

• Architect, build, and scale automated test frameworks within Databricks using PySpark, Python, and SQL.
• Design automated assertions for Delta Lake tables including data drift, schema evolution, and time-travel validation.
• Test large-scale batch and real-time streaming pipelines (Structured Streaming) for source-to-target integrity.
• Programmatically verify data lineage, audit logs, and Unity Catalog access controls.
• Lead CI/CD integration of data quality tests using Azure DevOps or GitHub Actions and Databricks Workflows.
• Mentor junior members, establish QA standards, and promote data quality across teams.
• Design and run automated performance and scalability tests on Spark jobs and clusters.

Skills

PySpark

Python

Spark SQL

Great Expectations

pytest

Delta Live Tables

Databricks

SDET

Data QA

CI/CD

Education

Bachelor’s or Master’s degree in Computer Science, Data Engineering

Tools

Databricks

Azure DevOps

GitHub Actions

Job description

V4C.ai in the United States seeks a Senior Data QA Automation Engineer to lead the quality strategy for our Databricks Lakehouse platforms, ensuring data quality across Delta Lakes, ETL pipelines, and enterprise data models.

You will mentor engineers, design scalable test frameworks, and own end-to-end data validation, governance verification, and CI/CD integration within Azure DevOps or GitHub Actions. This role drives data quality culture and engineering excellence.