Senior Data QA-Onshore

V4C.ai; United States; Remote; USD 140,000 - 190,000; Full time

Job summary

V4C.ai in the United States seeks a Senior Data QA Automation Engineer to lead the quality strategy for our Databricks Lakehouse platforms, ensuring data quality across Delta Lakes, ETL pipelines, and enterprise data models.

You will mentor engineers, design scalable test frameworks, and own end-to-end data validation, governance verification, and CI/CD integration within Azure DevOps or GitHub Actions. This role drives data quality culture and engineering excellence.

Qualifications

• 8+ years in data engineering, data QA, or SDET.
• 2+ years architecting test automation in Databricks.
• Proficient in PySpark, Python, and SQL.

Responsibilities

• Architect, build, and scale automated test frameworks within Databricks using PySpark, Python, and SQL.
• Design automated assertions for Delta Lake tables including data drift, schema evolution, and time-travel validation.
• Test large-scale batch and real-time streaming pipelines (Structured Streaming) for source-to-target integrity.
• Programmatically verify data lineage, audit logs, and Unity Catalog access controls.
• Lead CI/CD integration of data quality tests using Azure DevOps or GitHub Actions and Databricks Workflows.
• Mentor junior members, establish QA standards, and promote data quality across teams.
• Design and run automated performance and scalability tests on Spark jobs and clusters.

Skills

PySpark

Python

Spark SQL

Great Expectations

pytest

Delta Live Tables

Databricks

SDET

Data QA

CI/CD

Education

Bachelor’s or Master’s degree in Computer Science, Data Engineering

Tools

Databricks

Azure DevOps

GitHub Actions

Job description

About Us:

v4c.ai was founded with a clear goal: to make data, AI, and machine learning accessible and impactful for every organization. As a Databricks partner, we deliver end-to-end solutions that transform complex data challenges into strategic outcomes.

Job Summary

We are seeking a Senior Data QA Automation Engineer to lead the quality strategy, design, and implementation of automated testing frameworks for our big data platforms. In this senior role, you will own the end-to-end data validation strategy within our Databricks Lakehouse architecture, ensuring high-quality, reliable, and compliant data across Delta Lakes, ETL pipelines, and enterprise data models. You will work closely with Data Engineering leadership to establish rigorous quality gates and mentor mid-to-junior engineers on data testing best practices.

Key Responsibilities

• Strategic Framework Design: Architect, build, and scale automated test frameworks from scratch natively within Databricks using PySpark, Python, and SQL.
• Lakehouse Quality Engineering: Design robust automated assertions for Delta Lake tables, including checking data drift, schema evolution, and historical data validation via time-travel functions.
• Enterprise Pipeline Testing: Code complex automated scenarios to validate large-scale batch and real-time streaming data pipelines (Structured Streaming), ensuring source-to-target integrity.
• Governance Validation: Programmatically verify data lineage, audit logs, and access controls implemented via Databricks Unity Catalog.
• CI/CD & DevOps Ownership: Lead the integration of automated data quality tests into enterprise CI/CD pipelines (e.g., Azure DevOps, GitHub Actions), leveraging Databricks Workflows, APIs, or Airflow.
• Technical Leadership & Mentorship: Act as the subject matter expert for data quality; mentor junior team members, establish QA standards, and advocate for data quality principles across engineering teams.
• Performance Assessment: Design and execute automated performance and scalability tests on Spark jobs, large clusters, and complex query optimizations.

Required Skills and Qualifications

• Education: Bachelor’s or Master’s degree in Computer Science, Data Engineering, or a related quantitative field.
• Experience: 8+ years of experience in data engineering, data QA, or software development engineering in test (SDET), with at least 2+ years of dedicated experience architecting test automation in Databricks.
• Expert PySpark & Python: Mastery of Python and PySpark (DataFrames and SQL APIs) for processing and profiling large datasets.
• Advanced Spark SQL: Deep expertise in writing advanced SQL queries, optimization techniques, and understanding Spark query execution plans.
• Advanced Testing Tooling: Hands-on mastery of big-data validation libraries (e.g., Great Expectations, pytest, Delta Live Tables expectations).
• Cloud Infrastructure: Strong operational knowledge of Databricks deployment on a major cloud provider (AWS, Azure, or GCP).

Preferred Qualifications

• Certifications: Databricks Certified Data Engineer Professional or Databricks Certified Machine Learning Professional.
• Streaming Expertise: Experience validating real-time event-streaming architectures (Kafka, Event Hubs, Kinesis).
• Data Ops: Solid understanding of DataOps culture, testing infrastructure as code, and data observability principles.