ML Engineer - LLM & Agentic Evaluations

Turing

United States

Remote

USD 34,000 - 55,000

Full time

14 days+

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Benefits offered by this job

Fully remote work

Job summary

Turing is hiring ML Engineers to join its AI Evaluation team as contractors. You will design and run evaluation suites for LLMs, benchmark agentic workflows, and analyze results to improve frontier AI systems.

This fully remote role requires strong Python skills, experience with LLM APIs, and the ability to work across time zones with PST overlap. Engagement is a contractor assignment for a duration of 3 months, with a minimum of 20 hours per week and overlap of at least 5 hours per day with

Qualifications

  • Currently pursuing or recently completed a Bachelor's, Master’s, or PhD in Computer Science, Machine Learning, Data Science, Mathematics, or a related quantitative field.
  • Strong programming skills in Python, including NumPy, Pandas, Jupyter.
  • Working understanding of how modern LLMs are trained, prompted, and evaluated.
  • Experience interacting with LLM APIs (OpenAI, Anthropic, Google, Hugging Face, vLLM).
  • Strong analytical and writing skills—ability to document tasks, results, and recommendations.
  • Comfortable with fully remote work and overlapping collaboration hours.

Responsibilities

  • Design, build, and run evaluation suites for LLMs across reasoning, coding, math, tool use, and long-context tasks.
  • Develop benchmarks and rubrics for agentic workflows, including multi-step planning and tool calling.
  • Curate, clean, and annotate evaluation datasets; write clear task specs and grading criteria.
  • Run model comparisons and analyze results to produce structured reports with actionable insights.
  • Implement automated graders and validate them against human judgments.

Skills

Python
NumPy
Pandas
Jupyter
LLM APIs
Analytical writing
Remote work

Education

Bachelor's/Master's/PhD in CS/ML/Data Science

Tools

LangGraph
AutoGen
CrewAI
OpenAI Agents SDK

Job description

About Turing

Based in San Francisco, California, Turing is the world’s leading research accelerator for frontier AI labs and a trusted partner for global enterprises deploying advanced AI systems. Turing supports customers in two ways: first, by accelerating frontier research with high-quality data, advanced training pipelines, plus top AI researchers who specialize in coding, reasoning, STEM, multilinguality, multimodality, and agents; and second, by applying that expertise to help enterprises transform AI from proof of concept into proprietary intelligence with systems that perform reliably, deliver measurable impact, and drive lasting results on the P&L

Role Overview

We are hiring ML Engineers to join our AI Evaluation team, where you will help design and execute evaluations for Large Language Models (LLMs) and agentic AI workflows. You will work alongside ML engineers, researchers, and domain experts to measure model capabilities, surface failure modes, and improve the way frontier AI systems are trained and benchmarked. This is a hands-on, research-flavored role. You will work on real evaluation problems that directly influence how leading AI models are trained, refined, and shipped

What does day-to-day life look like? Design, build, and run evaluation suites for LLMs across reasoning, coding, math, tool use, and long-context tasks. Develop benchmarks and rubrics for agentic workflows, including multi-step planning, tool/function calling, code execution, and browser/computer-use agents. Curate, clean, and annotate evaluation datasets; write clear task specifications and grading criteria. Run model comparisons (open-source and frontier models), analyze results, and produce structured reports with actionable insights. Implement automated graders (programmatic checks, LLM-as-a-judge) and validate them against human reference judgments. Investigate failure cases, characterize error patterns, and write reproducible findings for the research team. Collaborate with engineers to integrate evaluations into internal harnesses, CI pipelines, and dashboards. Contribute to internal documentation, best practices, and tooling improvements.

Requirements
  • Currently pursuing or recently completed a Bachelor's, Master's, or PhD in Computer Science, Machine Learning, Data Science, Mathematics, or a related quantitative field.
  • Strong programming skills in Python, including familiarity with the standard data and ML stack (NumPy, Pandas, Jupyter, etc.).
  • Working understanding of how modern LLMs are trained, prompted, and evaluated.
  • Experience interacting with LLM APIs (OpenAI, Anthropic, Google, open-source via Hugging Face / vLLM, etc.).
  • Strong analytical and writing skills — ability to clearly document tasks, results, and recommendations.
  • Comfortable working independently in a fully remote environment with overlapping collaboration hours.
Nice to Have
  • Prior experience designing or running LLM evaluations or benchmarks (e.g., MMLU, GPQA, SWE-bench, GAIA, AgentBench, BIG-bench, HumanEval, TerminalBench, WebArena).
  • Hands-on experience with AI agent frameworks (LangGraph, AutoGen, CrewAI, OpenAI Agents SDK, etc.) or building tool-using agents from scratch.
  • Familiarity with LLM-as-a-judge methodologies, inter-annotator agreement, and statistical analysis of model outputs.
  • Experience with prompt engineering, few-shot design, and structured output formats (JSON Schema, function calling).
  • Exposure to RLHF / RLAIF, fine-tuning, or post-training data pipelines.
  • Background in software engineering, data annotation tooling, or research publications in NLP / ML.
Perks of Freelancing With Turing
  • Work in a fully remote environment.
  • Opportunity to work on cutting-edge AI projects with leading LLM companies.
Offer Details
  • Commitments Required: Minimum 20 hours per week with overlap of at least 5 hours/day with PST.
  • Engagement Type: Contractor assignment (no medical/paid leave)
  • Duration of Contract: 3 months (adjustable based on engagement)
  • Location: LATAM
Evaluation Process
  • Technical Interview
Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior Python Engineer – LLM Evaluation (US-based)
Senior Python Engineer – LLM Evaluation (US-based)

AI Trainer Gigs • California

Hybrid
USD 55,000 - 110,000
Senior Software Engineer – LLM Evaluation (US-based)
Senior Software Engineer – LLM Evaluation (US-based)

AI Trainer Gigs • United States

Remote
USD 83,000 - 165,000
Senior Software Engineer – LLM Evaluation
Senior Software Engineer – LLM Evaluation

Turing • United States

Remote
USD 55,000 - 96,000
Flexible engagement (10–40 hrs/week)
Contractor (no medical/paid leave)
Software Engineer – AI Code Evaluation & Benchmarking (US candidates only)
Software Engineer – AI Code Evaluation & Benchmarking (US candidates only)

Turing • United States

Remote
USD 83,000 - 152,000
Fully remote
Cutting-edge AI projects
Engineering Manager
Engineering Manager

Turing • United States

Remote
USD 83,000 - 138,000
Senior Staff Engineer (Machine Learning) - 45391
Senior Staff Engineer (Machine Learning) - 45391

Turing • Town of Italy (NY)

Remote
USD 120,000 - 160,000
Fully remote environment
Opportunity to work on cutting-edge AI projects
Flexible working hours
Senior Software Engineer – LLM Evaluation (US-based)
Senior Software Engineer – LLM Evaluation (US-based)

AI Trainer Gigs • San Francisco (CA)

On-site
USD 83,000 - 138,000
Senior Software Engineer – LLM Evaluation
Senior Software Engineer – LLM Evaluation

HumanitApp • California

On-site
USD 83,000 - 124,000
Senior Python Engineer – LLM Evaluation (US-based)
Senior Python Engineer – LLM Evaluation (US-based)

AI Trainer Gigs • United States

On-site
USD 83,000 - 138,000
Senior Software Engineer - 35501
Senior Software Engineer - 35501

Turing • New York (NY)

Remote
USD 68,880 - 206,640