Software Engineer – AI Agent Evaluation

Workada

United States

Remote

USD 90,000 - 140,000

Full time

37 hours ago
Be an early applicant

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

Workada is seeking a contributor who can convert real-world problems into self-contained Linux-terminal tasks for AI agents. You will craft briefs, set up environments with necessary files and tooling, build reference solutions, and implement automated checks to ensure correctness.

You’ll leverage deep knowledge of algorithms, OS, and data processing to identify failure modes and edge cases, communicating clearly where a workflow breaks and how to fix it.

Qualifications

  • PhD in CS or software engineering or 5+ years of professional experience in related field.
  • Strong Linux terminal and scripting skills (Python or Bash).
  • Experience with Docker and distributed/OS concepts is a plus.

Responsibilities

  • Translate problems into self-contained tasks for AI agents in a Linux terminal.
  • Create briefs, environment setups, reference solutions, and automated checks for each Task.
  • Identify and explain failure modes and edge cases in workflows.

Skills

Linux terminal
Python/Bash scripting
Concurrency and debugging
Research and problem solving

Education

PhD in computer science or software engineering

Tools

Docker
CI/CD pipelines

Job description

What You’ll Do

As a contributor, you’ll take problems from your own work and turn each one into a self-contained Task that an AI agent has to solve in a Linux terminal. For every Task, you’ll write a clear brief, set up the environment with the files and tools the agent needs, build a reference solution, and write automated checks on the final result. The strongest Tasks are the ones today’s best agents still get wrong. Your firsthand knowledge of where real work in your field goes wrong is what makes them valuable. That might be a race condition that only shows up under load, a migration that silently drops rows, a compiler pass that miscompiles an edge case, a consensus protocol that splits on a network partition, or a CI build that flakes because of test ordering.

Specialties We’re Looking For

We’re building depth across: Algorithms and data structures Distributed and operating systems , including concurrency and consensus Databases and SQL , including migrations and query planning Data engineering : pipelines, ETL, and streaming Compilers and programming languages Frontend and web DevOps, CI/CD, and build systems

You’re a Great Fit If

You hold, or are completing, a PhD in computer science or software engineering or a closely related field, or you have 5+ years of professional experience in it. You’ve done technical or computational work in your field and are comfortable in a Linux terminal: shell commands, scripting in Python or Bash, and installing packages. You’ve used Docker, or can pick it up quickly. We share templates and worked examples. You can draw on your own research, projects, code, or files as the source material for your Tasks. You have a precise, adversarial eye; you enjoy finding the exact point where a workflow breaks and explaining it cleanly. You’re honest about uncertainty and write instructions that leave no room for misreading. You’re based in the US or Canada. Academics and practitioners are equally welcome. No AI or machine-learning background is needed. Your expertise and your comfort working in a terminal are what we’re after.

Materials You Can Use

Almost any piece of your work can seed a Task, for example: Repositories, schemas, and failing test suites CI configs and build scripts Data pipelines and the datasets behind them Profiler traces, logs, and incident write-ups Pull requests and bug fixes from your own work Bugs, edge cases, and failure modes you’ve had to debug yourself

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

AI Agent Evaluation Expert
AI Agent Evaluation Expert

Workada • United States

Remote
USD 120,000 - 180,000
Security Engineer – AI Agent Evaluation
Security Engineer – AI Agent Evaluation

Workada • United States

Remote
USD 120,000 - 180,000
Finance & Operations Research Expert – AI Agent Evaluation
Finance & Operations Research Expert – AI Agent Evaluation

Workada • United States

Remote
USD 120,000 - 180,000
Life Sciences PhD – AI Agent Evaluation
Life Sciences PhD – AI Agent Evaluation

Workada • United States

Remote
USD 90,000 - 150,000
Music & Design Expert – AI Agent Evaluation
Music & Design Expert – AI Agent Evaluation

Workada • United States

Remote
USD 90,000 - 130,000
AI Evaluation Architect for Linux Task Challenges
AI Evaluation Architect for Linux Task Challenges

Workada • United States

Remote
USD 120,000 - 180,000
Research Engineer
Research Engineer

Klavis AI • San Francisco (CA)

On-site
USD 140,000 - 230,000
AI Agent Evaluation Engineer — Linux & Edge-Case Debugging
AI Agent Evaluation Engineer — Linux & Edge-Case Debugging

Workada • United States

Remote
USD 90,000 - 140,000
Task Designer, Software Engineering
Task Designer, Software Engineering

Reinforced Learning • Town of Tonawanda (NY)

On-site
USD 95,000 - 140,000
AI Trainer – BIOLOGY
AI Trainer – BIOLOGY

Planet Pharma • San Francisco (CA)

Remote
USD 120,000 - 180,000