Data Quality Engineer

SmartRecruiters, Inc.; Plano (TX); On-site; USD 95,000 - 125,000; Full time

Job summary

SmartRecruiters, Inc. seeks a Data Quality Engineer to design and run the data quality layer for a Databricks lakehouse using DQX. You will translate legacy business rules into automated checks and quarantine bad records before reaching Gold tables, providing visibility into data health.

You will define DQX checks, embed them in Databricks Jobs and pipelines, and build dashboards and alerts to track quality and SLA breaches.

Qualifications

• 3+ years in data engineering or data quality roles.
• Hands-on PySpark and Spark SQL on Databricks.
• Experience with DQX or comparable frameworks (Great Expectations, Soda, Deequ).
• Delta Lake and Unity Catalog fundamentals.
• Understanding data quality dimensions and measurement.
• Git-based workflow and CI/CD basics.

Responsibilities

• Profile data quality layer using the DQX profiler.
• Define DQX checks in YAML/JSON config or code.
• Set criticality per rule and implement quarantine patterns for invalid rows.
• Embed DQX checks into Databricks Jobs and Lakeflow (DLT) pipelines.
• Store check definitions and results in Delta tables; build DQ dashboards and alerts.
• Define DQ thresholds that act as quality gates (Bronze → Silver → Gold).
• Version-control rules and deploy them through CI/CD.

Skills

PySpark

Spark SQL

Databricks DQX

Delta Lake

Unity Catalog

CI/CD

Git workflow

Data quality concepts

Tools

Great Expectations

Soda

Deequ

Databricks

Job description

Position: Data Quality Engineer (Databricks DQX)

Job Type- 12-months contract

Job Description

Design and run the data quality layer of our new Databricks lakehouse using Databricks Labs DQX. You will turn business rules from the legacy SQL platform into automated checks, quarantine bad records before they reach Gold tables, and give the business visibility into data health.

Key Responsibilities

• Profile source and Bronze data with the DQX profiler and turn the output into reviewed quality rules.
• Define DQX checks (completeness, uniqueness, validity, ranges, referential and custom SQL checks) in YAML/JSON config or code.
• Set criticality per rule (error vs warn) and implement split/quarantine patterns for invalid rows.
• Embed DQX checks into Databricks Jobs and Lakeflow (DLT) pipelines for batch and streaming data.
• Store check definitions and results in Delta tables under Unity Catalog; build DQ dashboards and alerts.
• Define DQ thresholds that act as quality gates for layer promotion (Bronze → Silver → Gold).
• Version-control rules and deploy them through CI/CD.

Qualifications

Must-have skills :

• 3+ years in data engineering or data quality roles.
• Hands-on PySpark and Spark SQL on Databricks.
• Practical experience with DQX, or strong experience with a comparable framework (Great Expectations, Soda, Deequ, DLT expectations) and willingness to ramp up on DQX quickly.
• Delta Lake and Unity Catalog fundamentals.
• Understanding of data quality dimensions and how to measure them.
• Git-based workflow and CI/CD basics.

Nice-to-have skills :

• Azure Data Factory orchestration of Databricks jobs.
• Databricks SQL dashboards, Lakehouse Monitoring or Power BI for DQ reporting.
• Experience with SQL Server to cloud migrations.

Responsibilities

• DQX checks live on every Silver and Gold table in scope.
• Invalid records quarantined with a documented triage process.
• ADQ dashboard the business uses to track trends and SLA breaches.

Additional Information

All your information will be kept confidential according to EEO guidelines.