Applied AI Lead - Evaluation & Measurement

Leonardo DRS; Frederick (MD); Hybrid; USD 150,000 - 230,000; Full time

Benefits offered by this job

9/80 work schedule

Health benefits and insurance

Health savings account contribution

Paid vacation and leave

Flexible work schedules

Job summary

Leonardo DRS, Inc. AIS unit seeks an Applied AI Lead to design and oversee evaluation and measurement for AI systems across signals intelligence, electronic warfare, and mission systems.

This high-impact, individual-contributor role reports to senior leadership and shapes how we prove AI value before deployment. You will own the testing framework, define acceptance criteria, build measurement pipelines, and ensure traceability to ground truth.

Qualifications

• 5+ years in data, ML, or test engineering with 3+ years in evaluation or model validation.
• Hands-on large-language-model evaluation experience.
• Ability to design rigorous evaluation methodologies and measurement pipelines.

Responsibilities

• Define evaluation methodologies and gate criteria for ship readiness.
• Build measurement pipelines and maintain records of capability delivery.
• Link measurements to ground-truth data Source systems.
• Define logs for deployed systems to ensure diagnosability.
• Deliver monthly and quarterly results to Finance with accuracy.
• Manage AI idea intake and prevent work from going off-book.
• Set scope boundaries and create auditable proofs for claims.

Skills

Data evaluation

ML evaluation

Model validation

Test engineering

Communication

Education

Bachelor’s degree

Job description

Select how often (in days) to receive an alert:

Applied AI Lead - Evaluation & Measurement

Location: Frederick, MD, US, 21703 Fort Walton Beach, FL, US, 32548 Beavercreek, OH, US, 45431

Job ID:115203

The Leonardo DRS Airborne and Intelligence Systems business is a global leader and strategic partner committed to delivering world-class, full life-cycle defense and intelligence products that protect the security of our nation and our allies. From air combat training to state-of-the-art electronic warfare systems, our technology is deployed by virtually all U.S. military and government agencies around the world .

This is a hybrid position open to candidates who reside in or near Beavercreek, Ohio; Frederick, Maryland; or Fort Walton Beach, Florida.

Job Summary

Prove AI works. Prove what it's worth.

We've used AI before, now we treat it as a first-class citizen at Airborne & Intelligence Systems (AIS). We build and deploy AI across signals intelligence, electronic warfare, air combat training, and the mission systems that connect them. This work shows up on our floor, in the field, and on the company's P&L, while meeting the demanding standards of our defense and commercial customers. If you want your work and your talent to count — this is the place.

Two questions follow every AI system we build: does it actually work, reliably on real jobs and is it actually worth the money we say it is? You own both answers. You design the test every capability must pass before it can ship, and you build the measurement that turns results into honest numbers leadership can trust. You hold the line. Nothing ships until it clears your bar — no test, no launch — and you're deliberately independent from the person who builds and deploys, so the check is real. When there's pressure to over-claim a result, you're the one who holds the number honest.

At the heart of this function is a proprietary platform, the encoded way this business does AI. It takes any employee from a rough idea to a governed, well-formed use case, checks it against the rules, and prepares a recommendation a human decision-maker rules on. You don't just use it; you also build it. The quality bar and measurement logic you design get embedded into the platform itself, so the standard travels with the system to every team that touches it. You encode the acceptance standard and the value math once, and every team inherits both.

This is a high impact, founding individual-contributor seat within the technical core of AI Excellence function with meaningful end to end work ownership. You report directly to top leadership of the function giving you outsize visibility, opportunity for personal growth and direct line to impact. Your peers will be AI leads for Solutions & Delivery/ Infrastructure & Deployment

Job Responsibilities

Primary & Essential Accountabilities

• Define “good enough to ship.” Design the evaluation for each capability — baseline, acceptance criteria, error budget, human-review points — documented before launch. Gate the launch.
• Build the measurement. Stand up the pipelines and record keeping that track what each capability actually delivers.
• Establish the truth. Connect each measurement directly to the source system that holds the ground truth.
• Define what deployed systems must log. Set the trace and telemetry standard that keeps every shipped capability diagnosable and auditable.
• Produce the numbers. Deliver the monthly and quarterly results — forecast versus actual, clean and without double-counting — that the function defends to Finance.
• Keep the front door open. Run the record intake so every AI idea is captured, and nothing happens off the books.
• Scope boundaries. You are not the claim owner, governance or reporting. You build the instrument that proves or disproves the claims in an auditable manner.

Qualifications

• 5+ years in data, ML, or test engineering (regardless of total years exp), including 3+ years in evaluation or model validation, with recent hands-on large-language-model evaluation work.
• Demonstrated bi-directional evaluation rigor : an example where your evaluation stopped something from shipping, and one where it cleared something others doubted. Expect us to probe the methodology of each.
• Built a measurement practice where none existed. The first baselines, the first pipelines, the first dashboard leadership actually used, and it kept producing after you handed it off.
• Data and measurement instrumentation skills: pipelines, dashboards, record-keeping, baselines from source-of-record data.
• A demonstrated record of professional integrity under pressure: you have told someone, especially seniors “the number doesn’t support that claim” and held.
• Bachelor’s degree and 5 years experience or equivalent combination of both. Engineering, Computer Science, Data Science, or a related technical field is preferred
• Five years of relevant experience (or an equivalent combination of experience and training that provides the required knowledge, skills, and abilities
• Proficient technical expertise with demonstrated application
• Excellent interpersonal, leadership, negotiation, communication and writing skills

Preferred Qualifications

• Experience wiring metrics to operational systems of record (not survey- or estimate-based measurement).
• Statistical literacy for probabilistic acceptance: error budgets, confidence bounds, claims that survive challenge.
• Finance-adjacent exposure: cost accounting, benefits realization, or audit (a plus, not a requirement).
• Defense cost-accounting context.
• Active or prior security clearance.

U.S. Citizenship required. This position requires an active DOD security clearance or the ability to obtain such clearance within a reasonable time after commencement of employment.

Taking care of our people is a top priority at Leonardo DRS. We are proud to offer competitive salaries and comprehensive benefits, including medical, dental, and vision coverage, a company contribution to a health savings account, telemedicine, life and disability insurance, legal insurance, and a 401(k) savings plan.

• We champion wellness programs that focus on physical, emotional, and financial well-being.
• We develop our talent by offering programs and activities to support career-growth, professional development, and skill enhancement.
• And we understand there is more to life than work, and the importance of offering flexible work schedules with our 9/80 program, competitive vacation, health/emergency leave, paid parental leave, and community service hours.
• *Some employees are eligible for limited benefits only

Leonardo DRS, Inc. and its subsidiaries provide equal opportunities to all employees and applicants for employment and prohibit discrimination and harassment of any type without regard to race, color, religion, age, sex, national origin, disability status, protected veteran status, or any other characteristic protected by federal, state, or local laws. #AIS

Our Vision

To be the leading mid-tier defense technology company in the U.S.

Our Values

The Leonardo DRS culture is defined by our Core Values and Principles:

• Integrity
• Agility
• Excellence
• Customer Focus
• Community & Respect
• Innovation

We strive to uphold them in all aspects of our business practices to inspire our employees and provide outstanding support for our customers.