Applied AI Lead: Evaluation & Measurement (Hybrid)

Leonardo DRS; Frederick (MD); Hybrid; USD 150,000 - 230,000; Full time

Benefits offered by this job

9/80 work schedule

Health benefits and insurance

Health savings account contribution

Paid vacation and leave

Flexible work schedules

Job summary

Leonardo DRS, Inc. AIS unit seeks an Applied AI Lead to design and oversee evaluation and measurement for AI systems across signals intelligence, electronic warfare, and mission systems.

This high-impact, individual-contributor role reports to senior leadership and shapes how we prove AI value before deployment. You will own the testing framework, define acceptance criteria, build measurement pipelines, and ensure traceability to ground truth.

Qualifications

• 5+ years in data, ML, or test engineering with 3+ years in evaluation or model validation.
• Hands-on large-language-model evaluation experience.
• Ability to design rigorous evaluation methodologies and measurement pipelines.

Responsibilities

• Define evaluation methodologies and gate criteria for ship readiness.
• Build measurement pipelines and maintain records of capability delivery.
• Link measurements to ground-truth data Source systems.
• Define logs for deployed systems to ensure diagnosability.
• Deliver monthly and quarterly results to Finance with accuracy.
• Manage AI idea intake and prevent work from going off-book.
• Set scope boundaries and create auditable proofs for claims.

Skills

Data evaluation

ML evaluation

Model validation

Test engineering

Communication

Education

Bachelor’s degree

Job description

Leonardo DRS, Inc. AIS unit seeks an Applied AI Lead to design and oversee evaluation and measurement for AI systems across signals intelligence, electronic warfare, and mission systems.

This high-impact, individual-contributor role reports to senior leadership and shapes how we prove AI value before deployment. You will own the testing framework, define acceptance criteria, build measurement pipelines, and ensure traceability to ground truth.