AI Data Engineer — Build AI-Ready Data Pipelines

Howard Hughes Medical Institute (HHMI); Chevy Chase, Northern (MD, KY); Hybrid; USD 129,000 - 161,000; Full time

Benefits offered by this job

Hybrid schedule

Competitive pay and benefits

Job summary

Howard Hughes Medical Institute (HHMI) is seeking an AI Data Engineer to build and operate production data pipelines that support AI-driven workflows across HHMI’s administrative and research-facing platforms. You will own end-to-end data engineering, from ingestion to governance, leveraging Databricks, Delta Lake, and Unity Catalog to deliver governed, AI-ready data.

The role emphasizes hands-on pipeline development, medallion architecture, and retrieval-oriented infrastructure, collaborating

Qualifications

• Hands-on production data engineering experience with owning pipelines from development to production.
• Deep experience with Delta Lake, medallion architecture, Delta Live Tables, Workflows, and Unity Catalog.
• Proficiency in PySpark and SQL at scale; Git and CI/CD for data pipelines.
• Experience building AI-facing data foundations—embedding pipelines, vector stores, and retrieval evaluation.

Responsibilities

• Ingest data from landing zones and transform data from raw to bronze to silver to gold.
• Implement medallion architecture as pipelines, tables, and materialization schedules.
• Own the workflow-orchestration framework (Databricks Workflows, Delta Live Tables).
• Implement governance patterns including comprehensive Unity Catalog and access controls.
• Build and operate embedding pipelines and retrieval-supporting infrastructure.
• Collaborate with AI developers and operations teams on data contracts and quality checks.
• Develop and reuse reference-pattern libraries for pipelines and templates.
• Contribute to and evolve platform data pipelines to support AI and knowledge-management use cases.

Skills

Databricks Spark

Python SQL

Workflow orchestration

Data quality

Governance

AWS basics

Education

Bachelor’s degree or equivalent

Tools

Databricks

Unity Catalog

Delta Lake

Terraform

Job description

Howard Hughes Medical Institute (HHMI) is seeking an AI Data Engineer to build and operate production data pipelines that support AI-driven workflows across HHMI’s administrative and research-facing platforms. You will own end-to-end data engineering, from ingestion to governance, leveraging Databricks, Delta Lake, and Unity Catalog to deliver governed, AI-ready data.

The role emphasizes hands-on pipeline development, medallion architecture, and retrieval-oriented infrastructure, collaborating