Research Engineer - Heinz College

Carnegie Mellon University; Pittsburgh (Allegheny County); On-site; USD 90,000 - 120,000; Full time

Benefits offered by this job

Medical insurance

Dental and vision insurance

Tuition benefits

Paid time off

Pittsburgh bus pass

Fitness center access

Job summary

Carnegie Mellon University’s Heinz College seeks a Research Engineer to join the staff and advance agentic AI evaluation. You will help build computational infrastructure for two projects, emphasizing single-agent systems and developing models, benchmarks, and metrics.

The role requires collaboration with campus faculty, strong Python engineering skills, and the ability to conduct independent and team-driven work in a research setting.

Qualifications

• Master’s degree required; graduate-level education in CS/AI/ML or related field.
• Foundational understanding of LLMs and agentic AI systems, optimization, statistics, and machine learning.
• Strong Python engineering skills; at least one year of related experience.
• Demonstrated experience developing computational models, research software, or pipelines using Python.

Responsibilities

• Develop parts of a capability-aware evaluation framework for agentic AI systems, including capability detection, test recommendation, controlled execution, scoring, and reporting.
• Develop and maintain a versioned registry of AI evaluation metrics, benchmarks, datasets, scoring procedures, and reference results.
• Implement interfaces and research infrastructure for accessing and delivering evaluations, including MCP-based services and integration with established AI evaluation and sandboxing tools.
• Conduct benchmark experiments at the component, model, and end-to-end system levels, including controlled comparisons in which models are varied while the agent harness is held fixed.
• Assess the consistency and repeatability of evaluation methods by reproducing benchmarks, conducting empirical validation, and analyzing experimental outcomes.

Skills

Team collaboration

Independent work

Project management

Problem solving

Verbal & written communication

Python engineering

LLMs familiarity

Agentic AI systems

Education

Master’s degree

CS/AI/ML related field

Tools

Python

Job description

Job Overview

Carnegie Mellon University’s Heinz College is seeking a Research Engineer to join their staff. This opportunity is ideal for someone who flourishes in an engaging and challenging workplace. You will support the department by helping on two projects. Agentic AI evaluation: Build computational infrastructure and approaches to evaluate agentic AI systems, emphasizing single-agent systems at first. This involves capability detection, identifying and performing appropriate evaluations, handling benchmark and metric administration, managing controlled experiments, and reviewing performance across agent, model, and system layers. This project will develop mathematical models, optimization algorithms, and a pipeline. It will evaluate how workflows and roles change when AI is deployed. Across both projects, the successful candidate will provide technical contributions and play a key collaboration role. The candidate will drive research in partnership with campus faculty and other technical researchers from the CMU ecosystem.

Core Responsibilities

• Develop parts of a capability-aware evaluation framework for agentic AI systems, including capability detection, test recommendation, controlled execution, scoring, and reporting.
• Develop and maintain a versioned registry of AI evaluation metrics, benchmarks, datasets, scoring procedures, and reference results.
• Implement interfaces and research infrastructure for accessing and delivering evaluations, including MCP-based services and integration with established AI evaluation and sandboxing tools.
• Conduct benchmark experiments at the component, model, and end-to-end system levels, including controlled comparisons in which models are varied while the agent harness is held fixed.
• Assess the consistency and repeatability of evaluation methods by reproducing benchmarks, conducting empirical validation, and analyzing experimental outcomes.

Required Skills and Experience

• Works well in a team with students and faculty.
• Ability to conduct work independently and jointly with faculty supervisors.
• Strong project management skills.
• Strong problem-solving, reasoning, and analytical skills.
• Strong verbal and written communication skills.

Qualifications: Master’s degree required A combination of education and relevant experience from which comparable knowledge is demonstrated may be considered. Graduate-level education in computer science, artificial intelligence, machine learning, operations research, statistics, applied mathematics, engineering, or a related field. Foundational understanding of LLMs and agentic AI systems, optimization, statistics, and machine learning, together with strong Python engineering skills. At least one year of experience in any of these areas is preferred: LLM or agentic AI systems, AI evaluation and benchmarking, optimization modeling, simulation, statistical analysis, or AI-assisted software development. Graduate research, internships, or significant project work can meet this criterion. Demonstrated experience developing computational models, research software, or experimental pipelines using Python.

Requirements: Successful completion of a pre-employment background check.

Sponsorship

• Applicants for this position must be currently legally authorized to work for CMU in the United States.
• CMU will not sponsor or take over the sponsorship of an employment visa for this opportunity.
• Carnegie Mellon is not a qualifying employer for the STEM OPT benefit: only the 12-month OPT may be used to work at Carnegie Mellon.

Employment Details

• Fixed Term: This is a fixed-term position with an estimated duration of approximately seven (7) months ending on May 1, 2027.
• Typical Work Schedule: Monday-Friday from 8:30-5:00pm. An alternate work schedule allowing for evening and weekend hours will be required on occasion.
• Location: Pittsburgh, PA.
• Job Function: Researchers.
• Position Type: Staff – Fixed Term (Fixed Term) Full Time/Part time Full time.
• Pay Basis: Salary.
• More Information: Please visit “Why Carnegie Mellon” to learn more about becoming part of an institution inspiring innovations that change the world.

Benefits

Joining the CMU team opens the door to an array of exceptional benefits. Benefits eligible employees enjoy a wide array of benefits including comprehensive medical, prescription, dental, and vision insurance as well as a generous retirement savings program with employer contributions. Unlock your potential with tuition benefits, take well-deserved breaks with ample paid time off and observed holidays, and rest easy with life and accidental death and disability insurance. Additional perks include a free Pittsburgh Regional Transit bus pass, access to our Family Concierge Team to help navigate childcare needs, fitness center access, and much more! For a comprehensive overview of the benefits available, explore our Benefits page. At Carnegie Mellon, we value the whole package when extending offers of employment. Beyond credentials, we evaluate the role and responsibilities, your valuable work experience, and the knowledge gained from education and training. We appreciate your unique skills and the perspective you bring. Your journey with us is about more than just a job; it’s about finding the perfect fit for your professional growth and personal aspirations.

Carnegie Mellon University is an Equal Opportunity Employer/Disability/Veteran.

If your heart is in your work, come work with us.

Come join the architects of what’s next.

Please see Faculty Careers.

For technical assistance, email HR Services or call 412-268-4600. We are committed to providing an accessible experience throughout our recruitment process. If you need assistance or a reasonable accommodation at any stage of the application, interview, or hiring process, please contact Equal Opportunity Services by email at [e-mail] or by phone at 412-268-5072.