AI Agent Evaluation Expert

Workada; United States; Remote; USD 120,000 - 180,000; Full time

Job summary

Workada is looking for a meticulous contributor who can transform real-world problems into self-contained Tasks for AI agents in a Linux terminal. You will draft briefs, prepare the execution environment with tools and files, and build reference solutions with automated checks.

A strong candidate holds a PhD or 5+ years of experience in a relevant field, is proficient in Python or Bash, and can clearly document workflows for reproducibility.

Qualifications

• PhD in a relevant field or 5+ years of professional experience.
• Comfort working in a Linux terminal and scripting (Python or Bash).
• Experience with Docker and containerized workflows.
• Attention to detail and ability to explain workflows clearly.

Responsibilities

• Translate complex problems into self-contained Tasks with briefs and environment setup.
• Develop reference solutions and automated verification checks to validate outcomes.

Skills

Linux terminal

Python or Bash scripting

Docker

Adversarial problem solving

Technical communication

Education

PhD in a relevant field

Tools

Docker

Job description

What You'll Do

As a contributor, you'll take problems from your own work and turn each one into a self-contained Task that an AI agent has to solve in a Linux terminal. For every Task, you'll write a clear brief, set up the environment with the files and tools the agent needs, build a reference solution, and write automated checks on the final result.The strongest Tasks are the ones today's best agents still get wrong. Your firsthand knowledge of where real work in your field goes wrong is what makes them valuable. That might be a numerical solver that quietly loses precision, a migration that silently drops rows, a data loader that leaks the test split, a stripped binary hiding its real logic, a clock-domain crossing that passes simulation but fails in silicon, a reconciliation that's off by a currency conversion, or a color profile that shifts on conversion.

Specialties We're Looking For

If your background is in one of the fields below, there's likely a place for you. We're building depth across seven domains (among others):

• **Natural sciences and engineering:** biology, chemistry, physics, earth sciences, robotics, math, and linguistics
• **Software engineering:** algorithms, systems, databases, data engineering, frontend, and programming languages
• **Machine learning:** training, inference, evaluation, and kernels
• **Security:** cryptography, reverse engineering, forensics, and application security
• **Hardware:** CAD and mechanical design, and RTL and digital design
• **Business and finance:** finance, logistics, supply chain, claims, compliance, and marketing
• **Music and design:** composition, music technology, and audio production, and graphic, UX/UI, motion, and product design

You're a Great Fit If

• You hold, or are completing, a PhD in one of the fields above or a closely related field, or you have 5+ years of professional experience in one of them.
• You've done technical or computational work in your field and are comfortable in a Linux terminal: shell commands, scripting in Python or Bash, and installing packages.
• You've used Docker, or can pick it up quickly. We share templates and worked examples.
• You can draw on your own research, projects, code, or files as the source material for your Tasks.
• You have a precise, adversarial eye; you enjoy finding the exact point where a workflow breaks and explaining it cleanly.
• You're honest about uncertainty and write instructions that leave no room for misreading.
• You're based in the US or Canada.
• Academics and practitioners are equally welcome. Outside of the machine learning domain, no AI or machine-learning background is needed. Your expertise and your comfort working in a terminal are what we're after.

Materials You Can Use

Almost any piece of your technical work can seed a Task, for example:

• Scripts, code, and pipelines from your research or practice
• Datasets and the processing behind them
• Results, reports, or deliverables you can reproduce from raw sources
• Sequences, instrument data, and simulation configs
• Repositories, schemas, and failing test suites
• Model checkpoints, training code, and evaluation scripts
• Binaries, packet captures, and disk or memory images
• HDL designs, testbenches, and CAD models
• Ledgers, transaction exports, and SQL databases
• Audio, MIDI, notation, and design files
• Bugs, edge cases, and failure modes you've had to debug yourself