Data Platform Developer

Harley Ellis Devereaux (HED); Chicago (IL); Hybride; USD 110 000 - 140 000; Plein temps

Résumé du poste

Harley Ellis Devereaux (HED) is seeking a Data Platform Developer to build and mature our AI-enabled data lakehouse. You will own production pipelines, Databricks environment, and data governance, collaborating with analysts and developers to provide trusted analytics and AI-enabled workflows.

Join a hybrid work environment where you’ll shape platform health, document data assets, and drive continuous improvement while partnering with business data owners to ensure reliable, well-documented data

Qualifications

• At least 3 years of experience building and supporting production data pipelines on a cloud data platform.
• Strong SQL and Python skills; experience with Spark or a distributed processing framework.
• Practical experience troubleshooting production jobs and recovering or backfilling data.
• Experience implementing monitoring, alerting, and data reliability practices.
• Ability to write clear technical documentation and communicate with technical and business stakeholders.

Responsabilités

• Develop, operate, and improve ingestion and transformation pipelines across Bronze, Silver, and Gold layers; integrate Workday, Deltek, Dataverse, Microsoft Graph, and collaboration systems.
• Administer and improve the Databricks environment, including workflows, compute standards, access controls, and Unity Catalog governance.
• Maintain version-controlled, reproducible platform configurations and deployment workflows; resolve differences between source control and production.
• Implement automated data quality checks and schema contract tests to protect governed datasets.
• Monitor pipeline freshness, availability, performance, and cost; maintain alerts, dashboards, and runbooks.
• Investigate and resolve production incidents, coordinate data recovery and backfills, document root causes, and implement improvements.
• Maintain technical lineage, data dictionary, and platform docs; align with data owners on definitions and accountability.
• Assess downstream impacts of schema changes and coordinate communications with data consumers.
• Collaborate with analytics and AI teams to deliver reliable, well-governed data products and evaluate AI workflows.
• Contribute to platform architecture, technical standards, and onboarding of new data sources.

Connaissances

SQL

Python

Data pipelines

Operational incident response

Outils

Databricks

Spark

Unity Catalog

Delta Lake

Description du poste

Data Platform Developer

HED is hiring a Data Platform Developer to develop, evaluate, and improve the AI capabilities that support our people and our practice.

About HED

We are a team that is full of ideas, experience, creativity, passionate opinions, insatiable curiosity, uncompromising integrity, commitment, and skill. Our culture is about aspiration, embracing change and challenges, listening to (and learning from) each other, encouraging continual learning, and inspiring collective growth. As an inclusive, integrated architecture and engineering practice, we value the diversity of perspectives, experiences, abilities, and expertise that advance both the work we do, and the world we share.

Position Summary

As part of the Data & AI team, you will operate and improve a modern lakehouse built on Databricks, Delta Lake, and Unity Catalog, connecting business systems to trusted analytics and emerging AI applications.

This is an individual contributor role with meaningful ownership of production pipelines, platform health, data documentation, and continuous improvement. You will partner with analysts, application developers, and business data owners to make information accessible, dependable, and actionable. You will also help shape how AI-assisted tools are safely used to monitor and improve the platform.

Essential Functions

• Develop, operate, and improve ingestion and transformation pipelines across Bronze, Silver, and Gold data layers, integrating sources such as Workday, Deltek, Dataverse, Microsoft Graph, and collaboration systems.
• Administer and continuously improve the Databricks environment, including workflows, compute standards, data storage, access controls, and Unity Catalog governance.
• Maintain version-controlled, reproducible platform configurations and deployment workflows; identify and resolve differences between source control and production environments.
• Implement automated data quality checks and schema contract tests to protect governed datasets from unexpected changes.
• Monitor pipeline freshness, availability, performance, and cost; maintain actionable alerts, dashboards, and operational runbooks.
• Investigate and resolve production incidents, coordinate data recovery and backfills, document root causes, and implement preventive improvements.
• Maintain technical lineage, source metadata, the data dictionary, and platform documentation; collaborate with business owners to confirm definitions and accountability.
• Assess downstream impacts of schema changes, coordinate communications with data consumers, and support controlled changes across dependent systems.
• Collaborate with analytics and AI development colleagues to deliver reliable, well-governed data products and evaluate AI-assisted operational workflows.
• Contribute to platform architecture, technical standards, and future data-source onboarding as experience and ownership expand.

Requirements

• At least 3 years of experience building and supporting production data pipelines on a cloud data platform such as Databricks, Snowflake, Microsoft Fabric/Synapse, or BigQuery.
• Strong SQL and Python skills, with experience using Spark or a comparable distributed data processing framework.
• Practical experience troubleshooting production jobs, recovering or backfilling data, and participating in operational incident response.
• Experience implementing monitoring, alerting, and data reliability practices.
• Ability to write clear technical documentation and communicate effectively with technical colleagues and business stakeholders.
• Preferred: experience with Delta Lake or Apache Iceberg, medallion architectures, Databricks Workflows, Unity Catalog, and Azure Data Lake Storage.
• Preferred: familiarity with infrastructure as code, Git-based CI/CD, automated testing, data catalogs, lineage, and access governance.
• Preferred: experience with Grafana, Prometheus, Loki, cloud cost optimization, AI-assisted development, or enterprise systems such as Workday, Deltek, and Dynamics.
• Ability and interest to learn HED's specific tools and practices; direct experience with every technology listed is not required.

Physical Requirements

• Prolonged periods of sitting at a desk and working on a computer.
• Ability to communicate effectively in writing and verbally via phone, video conferencing, and in-person.
• Visual acuity sufficient to perform the responsibilities of the position, with or without reasonable accommodation.

Work Environment

HED embraces a hybrid work model that supports autonomy and collaboration, including opportunities to work from home and regular in-office time to connect with teammates and build culture.

Our offices provide a professional, open-space environment supporting both collaborative and independent work.

Other Duties

This job description is not intended to be a comprehensive listing of all activities, duties, and responsibilities required of the employee. Duties, responsibilities, and activities may change at any time, with or without notice.