Director of Engineering, IAEF

Digital Harbor Foundation

Baltimore (MD)

Remote

USD 215,000 - 315,000

Full time

7 days ago

Be an early applicant

Application generator

Don’t send a generic resume — generate a resume and cover letter tailored to this exact role.

Get past ATS filters

Benefits offered by this job

Health benefits & insurance

401k with matching

Paid time off

Parental leave

Wellness program

Job summary

Digital Harbor Foundation seeks a Founding Vice President, Engineering for the Independent AI Evaluation Foundation (IAEF). You will own platform architecture, delivery, and security leadership while guiding a distributed team and engaging with funders, partners, and executives.

You will shape evaluation platforms, HITL data capture, and partner with leaders to advance responsible AI evaluation at scale. This remote-first role prefers NYC or SF for in-person meetings, with travel ~15%.

Qualifications

Must have 12+ years of software engineering experience with 4+ years in senior technical leadership at a startup, evaluation/audit, ML infra, or research-grade platform.
Experience building/operating a platform with plugin/extension models and running AI evaluation, red-teaming, ML observability, or model-testing systems in production.
Proven track record hiring and retaining senior engineers in a distributed team, including ML or agentic-systems specialists.

Responsibilities

Own platform architecture and DTAP workflow components; integrate red-teaming, HITL data capture, and existing dashboards.
Recruit, develop, and retain engineering staff across backend/frontend/DevOps/Applied ML and product management.

Skills

Platform architecture

Team leadership

Security governance

ML tooling

Stakeholder communication

Hiring senior engineers

Job description

Overview

The Founding Vice President, Engineering, leads the architecture, delivery, and engineering team for Independent AI Evaluation Foundation (IAEF), a platform for evaluating the trustworthiness of artificial intelligence (AI). Reporting to the CEO of IAEF, this leader combines technical ownership, team development, security leadership, and external representation.

Independent AI Evaluation Foundation (IAEF)’s mission is to identify and reduce barriers to evaluating the efficacy, transparency, and safety of specific educational AI-centric solutions used by students and opportunity seekers in their educational context. IAEF is a fiscally sponsored project of Digital Harbor Foundation.

Digital Harbor Foundation is dedicated to fostering learning, creativity, productivity, and community through education with a vision of digital equity for everyone. Driven by our sincere belief that access to opportunity is a basic right, we take bold yet practical actions to support making a better future now. Through a portfolio of projects focused on developing leadership within communities, we support those closest to challenges to take deliberate actions based in design thinking approach, backed by data analysis, grounded in a practice of collective impact, and driven by a commitment to racial equity.

Education and Experience Requirements

Required

12+ years of software engineering experience, including at least four years in senior technical leadership at a startup, evaluation or audit firm, machine-learning infrastructure company, or research-grade platform team.
Experience building and operating a platform with a plugin, extension, or component model, and running AI evaluation, red-teaming, machine-learning observability, or model-testing systems in production.
A record of hiring and retaining senior engineers in a distributed team, including at least one applied machine-learning or agentic-systems specialist.

Strongly preferred

Prior CTO, VP Engineering, or founding-engineer experience at an early-stage company that raised institutional capital; experience enabling external developers to build on a platform; and prior evaluation-methodology work with academic or civil-society partners.
A PhD and extensive experience at a frontier AI research laboratory are not required.

Knowledge, Skills, and Abilities

Ability to design reusable platform components and work with human-in-the-loop (HITL) transcripts, benchmark scores, and large language model (LLM) red-teaming results.
Ability to communicate technical decisions to enterprise security leaders, foundation program officers, investors, customers, and partners.
Ability to prioritize competing technical and organizational needs and lead a distributed engineering team through delivery, operational issues, and growth.
Understanding of the operational separation required between a public benefit corporation, a companion nonprofit, and grant-funded programs.
Strongly preferred: experience leading implementation of ISO 42001, the NIST AI Risk Management Framework, or SOC 2 requirements, rather than participating only as a reviewer.

Additional Notes

This is a full-time, remote-first role based in the US or EU. New York or the San Francisco Bay Area is strongly preferred for quarterly in-person meetings with the CEO of IAEF, partners, and the board. Travel is approximately 15% for funder meetings, partner sites, conferences, and off‑sites.

Role and Responsibilities

Platform architecture and delivery

Own the Workbench’s Design → Test → Analyze → Publish (DTAP) workflow, Component Contract, Artifacts API, component registry, and canvas interface.
Create a persistent, structured, queryable evaluation history rather than one-off engagement exports, and integrate the existing red‑teaming platform, administrative dashboard, and HITL data‑capture architecture instead of rebuilding them from scratch.
Decide when to build or buy authentication, storage, orchestration, LLM-provider services, and sandboxing for the planned notebook environment.

Engineering leadership

In conjunction with the Human Resources Department of Digital Harbor Foundation, recruit, develop, and retain the engineering organization against the approved build plan, including Staff/Senior Backend, Senior Frontend, DevOps/site reliability, Applied Machine Learning, and Senior Product Manager roles
Set planning, code‑review, on‑call, incident‑response, and roadmap‑review practices; provide engineering progress and risks for board and funder reporting.

Evaluation platform strategy

Partner with the IAFE’s CEO and Evaluation Science Lead to translate HITL red teaming, automated benchmarks, LLM red teaming, LLM-as‑judge, and drift detection into a Component Contract that supports all three Test artifact types and planned agentic components.

Security, compliance, and trust

Lead vendor risk reviews for LLM providers, cloud infrastructure, and third‑party evaluation tools.
Coordinate with Digital Harbor Foundation’s Leadership, where applicable, on required approvals, funder restrictions, and separation of PBC and nonprofit resources, data, and intellectual property under approved agreements. Escalate unresolved risks rather than independently determining legal or sponsor policy.

External representation and fundraising

Represent IAEF in technical conversations with customers, partners, funders, and press, and own the technical content of investor fundraising decks, roadmap materials, demonstrations, and due diligence reviews in partnership with the Executive Director of IAEF and Digital Harbor Foundation’s leadership.
Lead the developer‑relations narrative for software and agent development kits (SDK/ADK) as the Component Contract opens to outside developers in Phase 1, partnering with the Developer Relations & Ecosystem hire planned for Period 2.

Other Duties

This job description is not a comprehensive listing of all activities, duties, or responsibilities required for this role. Duties, responsibilities, and activities may change at any time with or without notice.

Compensation

Compensation for this full‑time position is $215,000 - $315,000 annually, commensurate with experience.

Digital Harbor provides a best‑in‑class, comprehensive set of benefits to support the team. All regular, full‑time employees are eligible to receive:

Health Benefits & Insurance

Carefirst Blue Cross Blue Shield - Health, Dental, and Vision Insurance (100% of the premium paid for employees and 85% of dependents)
Pre‑Tax Health Savings Account (HSA) (with $365 monthly employer contributions)
Pre‑Tax Flexible Savings Account (FSA)
Paid Accidental Death & Dismemberment (AD&D) Insurance
Paid Short‑Term & Long‑Term Disability Insurance
Paid Basic Life Insurance
Supplemental Voluntary Life Insurance (Employee, Spouse & Dependent Children)
Total Pet Plan and Supplemental Wishbone Pet Insurance
Employee Opportunity Program (EAP) - Health and Wellness
Wellness Reimbursement Program

Retirement