AI Data Engineer

HHMI; United States; Híbrido; USD 120.000 - 150.000; Jornada completa

Ventajas ofrecidas por este puesto de trabajo

Hybrid work schedule

Descripción de la vacante

HHMI is seeking an AI Data Engineer to build and operate the data pipelines powering institutional AI applications. Based at our Chevy Chase HQ with a hybrid schedule, you will own end-to-end dataflow—from landing-zone ingestion through medallion-architecture transformations to AI-ready data products.

You will collaborate with the Principal Knowledge & Data Architect and AI Developers to ensure data quality, governance, and scalable delivery for product teams across HHMI.

Formación

• Hands-on production data engineering with pipelines owned end-to-end.
• Depth in Delta Lake medallion architecture and Databricks platforms.
• Fluent in Python and SQL for scalable ETL and analytics.

Responsabilidades

• Build AI-facing data pipelines from landing zone to AI-ready content.
• Implement medallion architecture with Delta Lake tables and schedules.
• Own the workflow-orchestration framework (Databricks Workflows, Delta Live Tables).
• Apply governance patterns via Unity Catalog, access control, and audits.
• Develop retrieval-supporting infra: embeddings, vector stores, reindexing.
• Partner with Operations Capabilities on data contracts and SLAs.
• Design data quality checks, monitoring, drift detection, alerts.
• Support AI Developer velocity with data provisioning for use cases.
• Contribute to reusable reference patterns and templates.

Conocimientos

Databricks

Delta Lake

PySpark

SQL

Python

Data pipelines

Herramientas

Databricks Workflows

Delta Live Tables

Unity Catalog

Descripción del empleo

Primary Work Address: 4000 Jones Bridge Road, Chevy Chase, MD, 20815

HHMI is focused on supporting and moving science forward in a variety of different ways ranging from conducting basic biomedical research, empowering educators, inspiring students, developing the next generation of scientists - even stretching into film and media production. Our Headquarters is in the greater Washington, DC metro area and is home to over 300 employees with expertise in investments , communications, digital production, biomedical sciences, and everything in between. The work housed here supports and augments the groundbreaking research conducted in HHMI labs across the nation. As HHMI scientists continue to push boundaries in laboratories and classrooms, you can be sure that your contributions while working here are making a difference.

The AI Accelerator exists to turn AI into daily reality across HHMI's administrative and operational functions. This role builds the data-engineering foundation every AI application and knowledge layer at HHMI runs on.

The work is hands-on. This person implements the pipelines, transformation patterns, and orchestration framework that turn HHMI's institutional content into governed, AI-ready data. They own the day-to-day execution of the data-engineering side of the AI fabric - pipeline development, medallion curation, workflow-orchestration framework, and governance implementation - under the design authority of the Principal Knowledge & Data Architect.

This role works closely with the Principal Knowledge & Data Architect (who owns the knowledge and retrieval layer's design), the AI Developer (who consumes what this role builds), and Operations Capabilities' Data Integration Engineer (who lands source-system data at the interface). The AI Data Engineer is the seat that connects those layers into working data flow.

HHMI's Principal K&D Architect designs the knowledge and data foundation for institutional AI, but the foundation only comes to life when someone builds and operates the pipelines that carry data through it. Without this role, our AI systems either wait on data that isn't ready or consume content whose quality can't be guaranteed. This role is what makes the K&D Architect's design real day-to-day - and it's the same role that keeps it working over years, as content evolves and models change.

This position works on a hybrid schedule, reporting in-person three days a week to our headquarters in Chevy Chase, Maryland.

Please note, we are not able to sponsor a visa for this position at this time.

What you will do

• Build the AI-facing data pipelines: ingestion from the landing zone Operations Capabilities delivers, transformation through raw → bronze → silver → gold, and serving of governed, AI-ready content for downstream consumption.
• Implement the medallion architecture: design patterns from the K&D Architect become working pipelines, tables, and materialization schedules. Delta Lake tables designed with partitioning, optimization, and evolution in mind.
• Own the workflow-orchestration framework: Databricks Workflows, Delta Live Tables, retry policies, alerting routes, run history, cost tags.
• Implement governance patterns: Unity Catalog structure, sensitivity classification, access control, and audit for AI-facing data assets. Design comes from the K&D Architect; day-to-day implementation lives here.
• Build and operate retrieval-supporting infrastructure: embedding pipelines, vector store maintenance, reindexing when models upgrade, retrieval evaluation frameworks.
• Partner with Operations Capabilities on source-system contracts: define what the AI Fabric consumes at the landing zone - schema, cadence, SLA, quality thresholds. Own the platform-side of that contract.
• Design and operate data quality: data-quality checks, freshness monitoring, drift detection, and the alerting that surfaces issues before they hit AI users.
• Support AI Developer velocity: when AI Developers deploy into product teams, this role is the data engineer they turn to when a use case needs specific data.
• Contribute to and consume the reference-pattern library: reusable pipeline patterns, code templates, and standards live in the shared platform layer. This role uses them, contributes new ones, and evolves them as we learn.

What We Are Looking For

• Hands-on production data engineering: at least four years designing, building, and operating production data pipelines. Not a Databricks-course track record - real production experience where you owned the pipeline through breakage, iteration, and recovery.
• Databricks and Spark depth: Delta Lake, medallion architecture, Delta Live Tables, Workflows, Databricks SQL, Unity Catalog. Comfortable at the layer where code meets platform.
• Python and SQL fluency: PySpark, ETL patterns, and SQL that runs at scale. Vers