Automation QA Lead – Data & AI Pipelines

TalentOla; Chicago (IL); Hybrid; USD 90,000 - 130,000; Full time

Job summary

TalentOla is seeking a data QA/automation engineer to lead automated testing for Databricks jobs, Delta tables, and ETL pipelines. You will validate data quality, reconciliation, and lineage across Databricks, Stardog, and Azure-based environments.

You will design test strategies, develop data validation utilities, and integrate automated tests into CI/CD workflows with Azure DevOps or GitHub Actions, collaborating in an Agile/DevOps setting.

Qualifications

• Experience with Databricks, Spark, Delta Lake and SQL.
• Proficiency in Python-based automation (e.g., Pytest) and CI/CD tooling.
• Knowledge of data QA, data reconciliation, data lineage and validation practices.

Responsibilities

• Develop automated test suites for Databricks jobs, Delta tables, views, and data transformations.
• Validate data quality rules and exception-handling processes.
• Create test data, validation queries, reusable utilities, and reconciliation frameworks.

Skills

Databricks

Spark

Delta Lake

SQL

Python

Pytest

CI/CD

Azure DevOps

GitHub Actions

Agile

Education

Engineering Degree BE/ME/BTech/MTech/BSc/MSc.

Tools

Stardog

SPARQL

Azure cloud services

ADLS

Kafka

Job description

TalentOla is seeking a data QA/automation engineer to lead automated testing for Databricks jobs, Delta tables, and ETL pipelines. You will validate data quality, reconciliation, and lineage across Databricks, Stardog, and Azure-based environments.

You will design test strategies, develop data validation utilities, and integrate automated tests into CI/CD workflows with Azure DevOps or GitHub Actions, collaborating in an Agile/DevOps setting.