Data Quality Engineer

Compunnel, Inc.; Chicago, Northern (IL, KY); Hybrid; USD 110,000 - 165,000; Full time

Job summary

Compunnel, Inc. is seeking an experienced Data Quality Engineer to design, implement, and operationalize enterprise-scale data quality frameworks within a modern Databricks Lakehouse architecture.

The ideal candidate will have strong expertise in Databricks, PySpark, Delta Lake, SQL, and data governance, with a track record of embedding data quality controls across the data lifecycle to ensure integrity, reliability, and observability of large cloud-based data platforms.

Qualifications

• Bachelor’s degree in Computer Science, Data Engineering, Information Systems, Data Science, or a related field.
• Experience in Data Engineering, Data Quality Engineering, or related data management roles.
• Strong hands-on experience with Databricks and PySpark.
• Extensive experience implementing enterprise data quality frameworks and controls within modern cloud data platforms.
• Advanced SQL development, data validation, and data profiling expertise.
• Experience integrating data quality processes into ELT/ETL pipelines and orchestration frameworks.

Responsibilities

• Design and implement enterprise-wide data quality frameworks aligned with Lakehouse architecture, including Bronze, Silver, and Gold data layers.
• Define, implement, and enforce data quality rules covering accuracy, consistency, timeliness, and validity.
• Develop reusable validation, reconciliation, profiling, and monitoring frameworks within Databricks environments.
• Establish automated data quality checks integrated into ELT and ETL pipelines.
• Embed quality controls into Databricks workflows, Spark pipelines, and Delta Lake architectures.
• Develop scalable validation processes supporting batch and real-time data ingestion.
• Partner with Data Engineers to enforce quality gates across ingestion, transformation, and consumption layers.
• Monitor data pipelines and identify anomalies, data drift, and quality degradation.

Skills

Databricks

PySpark

Delta Lake

SQL

Data governance

Cloud platforms

ELT/ETL

Education

Bachelor's degree in Computer Science or related field

Tools

Collibra

Deequ

Job description

We are seeking an experienced Data Quality Engineer to design, implement, and operationalize enterprise-scale data quality frameworks within a modern Databricks Lakehouse architecture. The ideal candidate will possess strong expertise in Databricks, PySpark, Delta Lake, SQL, and data governance, with a proven track record of embedding data quality controls throughout the data lifecycle. This role will be responsible for ensuring data integrity, reliability, compliance, and observability across large-scale cloud-based data platforms while driving quality-by-design principles across the organization.

KEY RESPONSIBILITIES

• Design and implement enterprise-wide data quality frameworks aligned with Lakehouse architecture, including Bronze, Silver, and Gold data layers.
• Define, implement, and enforce data quality rules covering:
  • - Accuracy
  • - Consistency
  • - Timeliness
  • - Validity
• Develop reusable validation, reconciliation, profiling, and monitoring frameworks within Databricks environments.
• Establish automated data quality checks integrated into ELT and ETL pipelines.
• Embed quality controls directly into Databricks workflows, Spark processing pipelines, and Delta Lake architectures.
• Develop scalable validation processes supporting both batch and real-time data ingestion pipelines.
• Partner with Data Engineers to ensure quality gates are enforced across ingestion, transformation, and consumption layers.
• Optimize data quality processes for performance, scalability, and reliability across large distributed datasets.
• Implement and maintain data observability solutions, including dashboards, alerts, monitoring metrics, and reporting frameworks.
• Monitor data pipelines and proactively identify anomalies, failures, data drift, and data quality degradation.
• Lead root cause analysis (RCA) activities and drive resolution of data quality issues.
• Develop and maintain enterprise data quality scorecards and performance reporting.
• Ensure adherence to enterprise data governance standards, including metadata management, data lineage, traceability, and auditability.
• Collaborate with Data Governance teams to align data definitions, ownership models, and control frameworks.
• Support regulatory compliance requirements through auditable and repeatable data quality processes.
• Define and enforce data quality SLAs, standards, and data contracts across business domains.
• Implement CI/CD practices for data quality rules, monitoring processes, and validation frameworks.
• Automate testing and validation of data transformations, integrations, and pipelines.
• Develop reusable enterprise libraries and frameworks for scalable data quality enforcement.
• Partner with Data Architects, BI teams, Data Engineers, and business stakeholders to drive data quality initiatives.
• Provide technical leadership, mentorship, and best-practice guidance across teams.
• Serve as the subject matter expert (SME) for enterprise data quality strategies and standards.
• Drive continuous improvement and innovation in data quality methodologies, tools, and practices.

REQUIRED QUALIFICATIONS

• Bachelor’s degree in Computer Science, Data Engineering, Information Systems, Data Science, or a related field.
• Proven experience in Data Engineering, Data Quality Engineering, or related data management roles.
• Strong hands-on experience with:
  • Databricks
  • PySpark
• Extensive experience implementing enterprise data quality frameworks and controls within modern cloud data platforms.
• Advanced SQL development, data validation, and data profiling expertise.
• Experience integrating data quality processes into ELT/ETL pipelines and orchestration frameworks.
• Strong knowledge of data lifecycle management principles and best practices.
• Experience working with large-scale datasets in AWS or Azure cloud environments.
• Strong understanding of:
  • Data Governance
  • Metadata Management
  • Data Quality Controls
  • Auditability Requirements
• Experience implementing automated validation, reconciliation, and monitoring processes.
• Strong analytical, troubleshooting, and problem-solving skills.
• Ability to translate business requirements into scalable technical solutions.
• Excellent communication and stakeholder management skills.
• Experience working within Agile and DevOps delivery environments.

PREFERRED QUALIFICATIONS

• Experience within Financial Services, Banking, Insurance, or other regulated industries.
• Familiarity with Collibra or other Enterprise Data Governance platforms.
• Experience implementing data observability and monitoring solutions, including:
  • Deequ
• Experience with real-time or streaming data quality validation frameworks.
• Knowledge of regulatory reporting, compliance controls, and data governance programs.
• Experience implementing data contracts and enterprise data standards.
• Exposure to machine learning-driven data quality monitoring solutions.
• Experience supporting enterprise Lakehouse modernization initiatives.

CERTIFICATIONS

• Databricks Certified Data Engineer Associate or Professional (Preferred)
• Databricks Certified Developer for Apache Spark (Preferred)
• AWS Certified Data Engineer – Associate (Preferred)
• Relevant Data Quality, Data Governance, or Cloud Certifications (Preferred)