Senior AI Evaluation & Benchmarking Engineer

Thomson Reuters; New York (NY); Hybrid; USD 147,000 - 273,000; Full time

Benefits offered by this job

Hybrid Work Model

Mental Health Days off

Headspace access

Tuition reimbursement

Employee stock purchase plan

Job summary

Thomson Reuters Labs is seeking a Lead Research Engineer to lead production-ready software for AI model evaluation and benchmarking. You will design scalable evaluation infrastructure, extend tools, and collaborate with researchers to compare LLMs and other AI models with production-grade Python software.

You will own the engineering approach from prototypes to reliable tools, mentoring engineers and aligning with product partners while ensuring traceable, cost-aware, and high-quality

Qualifications

• 8-10+ years of relevant software engineering experience, including building and maintaining production systems involving machine learning or complex data.
• Strong hands-on Python programming skills, including API design, automated testing, and working with structured and unstructured data.
• Technical leadership and end-to-end delivery experience, including architecture decisions, engineering planning, staffing and effort estimation.
• Experience collaborating with research scientists or machine learning practitioners to translate experiments and research methods into working software.
• Experience integrating or testing machine learning models, interpreting evaluation results, and applying findings to engineering decisions.
• Understanding of language models and the ability to recognize unreliable comparisons or test-data leakage.
• Cloud engineering experience using AWS, Azure, or a comparable platform, including distributed workflows, automated deployment, and monitoring.
• Working knowledge of SQL and experience developing software that processes structured and unstructured data.
• Strong collaboration, communication, problem-solving, and mentoring skills, with experience coordinating technical delivery across research, engineering, and product stakeholders.

Responsibilities

• Lead technical delivery and engineering planning: define priorities, break projects into stages, estimate effort and cost, staff appropriately.
• Design and build scalable evaluation infrastructure: extend tools, design new components, produce production-quality Python software.
• Turn research methods into working software: build automated evaluation pipelines connecting model APIs, data, and scorers.
• Ensure reliable, reproducible model comparisons: maintain repeatable experiments, keep evaluation data separate, trace results to inputs.
• Build practical tools for researchers and product teams: create interfaces, reports, and diagnostics for easy use.
• Own production reliability, performance, and responsible data use: integrate testing, deployment, monitoring, access controls.
• Collaborate, mentor, and guide engineering decisions: align interfaces, resolve dependencies, and communicate trade-offs.

Skills

Python programming

Technical leadership

Experimentation with ML

Cloud engineering

SQL

Communication & mentoring

Education

Bachelor's degree in Computer Science, Engineering, or related field

Job description

Thomson Reuters Labs is seeking a Lead Research Engineer to lead production-ready software for AI model evaluation and benchmarking. You will design scalable evaluation infrastructure, extend tools, and collaborate with researchers to compare LLMs and other AI models with production-grade Python software.

You will own the engineering approach from prototypes to reliable tools, mentoring engineers and aligning with product partners while ensuring traceable, cost-aware, and high-quality