Data Engineer - AI-Driven Pipelines & Quality

SCOR Group; Leawood (KS); On-site; USD 95,000 - 165,000; Full time

Job summary

SCOR in Leawood, KS is seeking an experienced Data Engineer to design, build, and operate data pipelines and analytics-ready datasets powering reporting and decision-making.

You will implement ETL/ELT processes with Python, Azure Databricks, and SQL across Microsoft SQL Server and PostgreSQL, while embedding responsible AI, governance, and quality checks into production data platforms.

Qualifications

• 5+ years in Data engineering roles.
• Strong Python for ETL, automation, and data transformations.
• Experience with Azure data services and Databricks operations.
• Proven experience with AI-assisted software development tools and workflows.
• Demonstrated ability to use generative AI to improve developer productivity and solution quality.
• Solid understanding of responsible AI principles, including data privacy, fairness, and explainability.
• Hands-on experience integrating AI capabilities into enterprise data or analytics applications.
• Degree in Computer Science, Management of Information Systems, or a related analytical field or equivalent experience.
• Nice-to-haves: data quality testing/validation frameworks; Delta tables; PostgreSQL optimization; CI/CD for data pipelines with Azure DevOps or GitLab; integrating AI/ML endpoints.

Responsibilities

• Design, build, and maintain batch and incremental data pipelines using Python, Azure Databricks, and SQL.
• Automate and schedule pipeline execution, implement idempotent processing, and build monitoring/alerting.
• Own day-to-day production support for data pipelines and reporting workflows with incident triage and root-cause analysis.
• Develop notebooks and jobs in Databricks; create reusable libraries and parameterized workflows; tune performance.
• Optimize SQL across Microsoft SQL Server and PostgreSQL; troubleshoot performance and ensure data integrity.
• Define validation rules, reconciliation checks, and automated tests to enforce SLAs and trust in analytics.
• Leverage AI-assisted development and generative AI to improve productivity and governance.

Skills

Python

ETL/ELT

Azure Databricks

SQL

AI tools

Responsible AI

SQL Server

PostgreSQL

Data pipelines

Education

CS degree

Tools

Azure DevOps

GitLab

Delta Lake

Job description

SCOR in Leawood, KS is seeking an experienced Data Engineer to design, build, and operate data pipelines and analytics-ready datasets powering reporting and decision-making.

You will implement ETL/ELT processes with Python, Azure Databricks, and SQL across Microsoft SQL Server and PostgreSQL, while embedding responsible AI, governance, and quality checks into production data platforms.