Databricks Data Quality Engineer — DQX & Dashboards

SmartRecruiters, Inc.; Plano (TX); On-site; USD 95,000 - 125,000; Full time

Job summary

SmartRecruiters, Inc. seeks a Data Quality Engineer to design and run the data quality layer for a Databricks lakehouse using DQX. You will translate legacy business rules into automated checks and quarantine bad records before reaching Gold tables, providing visibility into data health.

You will define DQX checks, embed them in Databricks Jobs and pipelines, and build dashboards and alerts to track quality and SLA breaches.

Qualifications

• 3+ years in data engineering or data quality roles.
• Hands-on PySpark and Spark SQL on Databricks.
• Experience with DQX or comparable frameworks (Great Expectations, Soda, Deequ).
• Delta Lake and Unity Catalog fundamentals.
• Understanding data quality dimensions and measurement.
• Git-based workflow and CI/CD basics.

Responsibilities

• Profile data quality layer using the DQX profiler.
• Define DQX checks in YAML/JSON config or code.
• Set criticality per rule and implement quarantine patterns for invalid rows.
• Embed DQX checks into Databricks Jobs and Lakeflow (DLT) pipelines.
• Store check definitions and results in Delta tables; build DQ dashboards and alerts.
• Define DQ thresholds that act as quality gates (Bronze → Silver → Gold).
• Version-control rules and deploy them through CI/CD.

Skills

PySpark

Spark SQL

Databricks DQX

Delta Lake

Unity Catalog

CI/CD

Git workflow

Data quality concepts

Tools

Great Expectations

Soda

Deequ

Databricks

Job description

SmartRecruiters, Inc. seeks a Data Quality Engineer to design and run the data quality layer for a Databricks lakehouse using DQX. You will translate legacy business rules into automated checks and quarantine bad records before reaching Gold tables, providing visibility into data health.

You will define DQX checks, embed them in Databricks Jobs and pipelines, and build dashboards and alerts to track quality and SLA breaches.