Hybrid Data Platform Engineer - AI Pipelines

Harley Ellis Devereaux (HED); Chicago (IL); Hybrid; USD 110,000 - 140,000; Full time

Job summary

Harley Ellis Devereaux (HED) is seeking a Data Platform Developer to build and mature our AI-enabled data lakehouse. You will own production pipelines, Databricks environment, and data governance, collaborating with analysts and developers to provide trusted analytics and AI-enabled workflows.

Join a hybrid work environment where you’ll shape platform health, document data assets, and drive continuous improvement while partnering with business data owners to ensure reliable, well-documented data

Qualifications

• At least 3 years of experience building and supporting production data pipelines on a cloud data platform.
• Strong SQL and Python skills; experience with Spark or a distributed processing framework.
• Practical experience troubleshooting production jobs and recovering or backfilling data.
• Experience implementing monitoring, alerting, and data reliability practices.
• Ability to write clear technical documentation and communicate with technical and business stakeholders.

Responsibilities

• Develop, operate, and improve ingestion and transformation pipelines across Bronze, Silver, and Gold layers; integrate Workday, Deltek, Dataverse, Microsoft Graph, and collaboration systems.
• Administer and improve the Databricks environment, including workflows, compute standards, access controls, and Unity Catalog governance.
• Maintain version-controlled, reproducible platform configurations and deployment workflows; resolve differences between source control and production.
• Implement automated data quality checks and schema contract tests to protect governed datasets.
• Monitor pipeline freshness, availability, performance, and cost; maintain alerts, dashboards, and runbooks.
• Investigate and resolve production incidents, coordinate data recovery and backfills, document root causes, and implement improvements.
• Maintain technical lineage, data dictionary, and platform docs; align with data owners on definitions and accountability.
• Assess downstream impacts of schema changes and coordinate communications with data consumers.
• Collaborate with analytics and AI teams to deliver reliable, well-governed data products and evaluate AI workflows.
• Contribute to platform architecture, technical standards, and onboarding of new data sources.

Skills

SQL

Python

Data pipelines

Operational incident response

Tools

Databricks

Spark

Unity Catalog

Delta Lake

Job description

Harley Ellis Devereaux (HED) is seeking a Data Platform Developer to build and mature our AI-enabled data lakehouse. You will own production pipelines, Databricks environment, and data governance, collaborating with analysts and developers to provide trusted analytics and AI-enabled workflows.

Join a hybrid work environment where you’ll shape platform health, document data assets, and drive continuous improvement while partnering with business data owners to ensure reliable, well-documented data