Agentic AI Research Engineer — Evaluation & Infrastructure

Carnegie Mellon University; Pittsburgh (Allegheny County); On-site; USD 90,000 - 120,000; Full time

Benefits offered by this job

Medical insurance

Dental and vision insurance

Tuition benefits

Paid time off

Pittsburgh bus pass

Fitness center access

Job summary

Carnegie Mellon University’s Heinz College seeks a Research Engineer to join the staff and advance agentic AI evaluation. You will help build computational infrastructure for two projects, emphasizing single-agent systems and developing models, benchmarks, and metrics.

The role requires collaboration with campus faculty, strong Python engineering skills, and the ability to conduct independent and team-driven work in a research setting.

Qualifications

• Master’s degree required; graduate-level education in CS/AI/ML or related field.
• Foundational understanding of LLMs and agentic AI systems, optimization, statistics, and machine learning.
• Strong Python engineering skills; at least one year of related experience.
• Demonstrated experience developing computational models, research software, or pipelines using Python.

Responsibilities

• Develop parts of a capability-aware evaluation framework for agentic AI systems, including capability detection, test recommendation, controlled execution, scoring, and reporting.
• Develop and maintain a versioned registry of AI evaluation metrics, benchmarks, datasets, scoring procedures, and reference results.
• Implement interfaces and research infrastructure for accessing and delivering evaluations, including MCP-based services and integration with established AI evaluation and sandboxing tools.
• Conduct benchmark experiments at the component, model, and end-to-end system levels, including controlled comparisons in which models are varied while the agent harness is held fixed.
• Assess the consistency and repeatability of evaluation methods by reproducing benchmarks, conducting empirical validation, and analyzing experimental outcomes.

Skills

Team collaboration

Independent work

Project management

Problem solving

Verbal & written communication

Python engineering

LLMs familiarity

Agentic AI systems

Education

Master’s degree

CS/AI/ML related field

Tools

Python

Job description

Carnegie Mellon University’s Heinz College seeks a Research Engineer to join the staff and advance agentic AI evaluation. You will help build computational infrastructure for two projects, emphasizing single-agent systems and developing models, benchmarks, and metrics.

The role requires collaboration with campus faculty, strong Python engineering skills, and the ability to conduct independent and team-driven work in a research setting.