Data Platform Engineer - AI-Driven Lakehouse

Harley Ellis Devereaux (HED); San Francisco (CA); Hybrid; USD 140,000 - 190,000; Full time

Benefits offered by this job

Hybrid work model

Job summary

Harley Ellis Devereaux (HED) is hiring a Data Platform Developer to build, evaluate, and improve the AI capabilities that support our people and practice. You will operate a modern lakehouse built on Databricks, Delta Lake, and Unity Catalog, linking business systems to trusted analytics.

As an individual contributor, you own production pipelines, platform health, and data documentation, partnering with analysts and developers to deliver well-governed data products.

Qualifications

• 3+ years building production data pipelines on cloud platforms such as Databricks or Snowflake.
• Strong SQL and Python skills with Spark or similar framework.
• Experience with monitoring, incident response, and data reliability practices.
• Ability to document technical details clearly for stakeholders.

Responsibilities

• Develop, operate, and improve data pipelines across Bronze, Silver, and Gold layers.
• Administer Databricks environment, workflows, and Unity Catalog governance.
• Maintain versioned, reproducible platform configurations and deployment workflows.
• Implement automated data quality checks and schema tests.
• Monitor pipeline freshness, performance, and cost; maintain alerts and runbooks.
• Investigate production incidents and coordinate data recovery/backfills.
• Maintain data dictionary, lineage, and platform documentation.
• Collaborate with analytics and AI teams to deliver governed data products.
• Contribute to platform architecture and future data-source onboarding.

Skills

SQL

Python

Spark

Data pipelines

Tools

Databricks

Unity Catalog

Delta Lake

Azure Data Lake Storage

Job description

Harley Ellis Devereaux (HED) is hiring a Data Platform Developer to build, evaluate, and improve the AI capabilities that support our people and practice. You will operate a modern lakehouse built on Databricks, Delta Lake, and Unity Catalog, linking business systems to trusted analytics.

As an individual contributor, you own production pipelines, platform health, and data documentation, partnering with analysts and developers to deliver well-governed data products.