Databricks Lakehouse Data Quality Engineer

Compunnel, Inc.; Chicago, Northern (IL, KY); Hybrid; USD 110,000 - 165,000; Full time

Job summary

Compunnel, Inc. is seeking an experienced Data Quality Engineer to design, implement, and operationalize enterprise-scale data quality frameworks within a modern Databricks Lakehouse architecture.

The ideal candidate will have strong expertise in Databricks, PySpark, Delta Lake, SQL, and data governance, with a track record of embedding data quality controls across the data lifecycle to ensure integrity, reliability, and observability of large cloud-based data platforms.

Qualifications

• Bachelor’s degree in Computer Science, Data Engineering, Information Systems, Data Science, or a related field.
• Experience in Data Engineering, Data Quality Engineering, or related data management roles.
• Strong hands-on experience with Databricks and PySpark.
• Extensive experience implementing enterprise data quality frameworks and controls within modern cloud data platforms.
• Advanced SQL development, data validation, and data profiling expertise.
• Experience integrating data quality processes into ELT/ETL pipelines and orchestration frameworks.

Responsibilities

• Design and implement enterprise-wide data quality frameworks aligned with Lakehouse architecture, including Bronze, Silver, and Gold data layers.
• Define, implement, and enforce data quality rules covering accuracy, consistency, timeliness, and validity.
• Develop reusable validation, reconciliation, profiling, and monitoring frameworks within Databricks environments.
• Establish automated data quality checks integrated into ELT and ETL pipelines.
• Embed quality controls into Databricks workflows, Spark pipelines, and Delta Lake architectures.
• Develop scalable validation processes supporting batch and real-time data ingestion.
• Partner with Data Engineers to enforce quality gates across ingestion, transformation, and consumption layers.
• Monitor data pipelines and identify anomalies, data drift, and quality degradation.

Skills

Databricks

PySpark

Delta Lake

SQL

Data governance

Cloud platforms

ELT/ETL

Education

Bachelor's degree in Computer Science or related field

Tools

Collibra

Deequ

Job description

Compunnel, Inc. is seeking an experienced Data Quality Engineer to design, implement, and operationalize enterprise-scale data quality frameworks within a modern Databricks Lakehouse architecture.

The ideal candidate will have strong expertise in Databricks, PySpark, Delta Lake, SQL, and data governance, with a track record of embedding data quality controls across the data lifecycle to ensure integrity, reliability, and observability of large cloud-based data platforms.