Databricks QA Automation Engineer - Data Integrity & CI/CD

RXinsider LTD.; Edina (MN); Hybrid; USD 89,000 - 156,000; Full time

Job summary

Vizient in Edina, MN, is seeking a data quality and test automation engineer to develop and maintain automated test suites for Databricks jobs, Delta tables, and ETL pipelines. You will validate metrics, data transformations, and ensure referential integrity across systems, collaborating with data engineering and platform teams.

You will build test data, SQL validations, and reconciliation frameworks, and you’ll integrate automated tests into CI/CD pipelines to raise the reliability of

Qualifications

• Experience building automated tests for data platforms.
• Experience validating large-scale data transformations and metrics.
• Strong understanding of data quality and reconciliation concepts.

Responsibilities

• Develop and maintain automated test suites for Databricks jobs, Delta tables, views, data transformations, and ETL pipelines.
• Validate metric calculations, business rules, aggregations, derived values, and source-to-target data transformations within Databricks.
• Perform source-to-target reconciliation and validate data completeness, accuracy, consistency, timeliness, and referential integrity across systems.
• Develop automated tests for full and incremental loads, updates, deletes, retries, exception handling, and recovery scenarios.
• Create test data, SQL validation queries, reusable utilities, and reconciliation frameworks to improve testing efficiency and coverage.
• Integrate automated data tests into CI/CD pipelines and enhance test automation frameworks and processes.
• Perform functional, integration, regression, and end-to-end testing across large-scale data solutions.
• Analyze test failures, conduct root cause analysis, document defects, and collaborate with data engineering and platform teams to resolve issues.

Skills

Databricks

Automated testing

CI/CD pipelines

Data validation

Tools

Stardog

SPARQL

SQL

Job description

Vizient in Edina, MN, is seeking a data quality and test automation engineer to develop and maintain automated test suites for Databricks jobs, Delta tables, and ETL pipelines. You will validate metrics, data transformations, and ensure referential integrity across systems, collaborating with data engineering and platform teams.

You will build test data, SQL validations, and reconciliation frameworks, and you’ll integrate automated tests into CI/CD pipelines to raise the reliability of