Software Engineer, Evaluation Tooling

Modulus

United States

Remote

USD 140,000 - 220,000

Full time

4 days ago
Be an early applicant

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Benefits offered by this job

Health insurance
Paid time off

Job summary

Modulus in the United States is seeking a talented engineer to build internal software that evaluates AI assistants and language models at scale. You will develop services to query models, preserve context for reproducibility, and enable researchers to compare results across models and versions.

The role is full-time and remote within the United States, with the option to work from Scottsdale, AZ. The position offers an annual salary range of $140,000–$220,000, health insurance, and paid time

Qualifications

  • Production software experience in TypeScript or Python.
  • Experience with data pipelines, APIs, and PostgreSQL.
  • Familiarity with language model APIs and nondeterministic outputs.
  • Care for correctness, testing, and reproducibility.

Responsibilities

  • Build services that query AI assistants and language models through APIs and other interfaces.
  • Create data pipelines that store outputs with enough context to reproduce each run.
  • Develop tools for comparing answers across models, versions, and time.
  • Build internal interfaces for running, reviewing, and annotating evaluations.
  • Keep tools reliable, tested, secure, and well documented.

Skills

TypeScript
Python
Data pipelines
APIs
Language Models

Tools

PostgreSQL

Job description

Role overview

Build internal software that runs evaluations of AI assistants and language models at scale. The work covers querying systems, preserving the context needed to reproduce their outputs, and helping research and analysis teams compare results across models and versions.

Responsibilities
  • Build services that query AI assistants and language models through APIs and other interfaces.
  • Create data pipelines that store outputs with enough context to reproduce each run.
  • Develop tools for comparing answers across models, versions, and time.
  • Build internal interfaces for running, reviewing, and annotating evaluations.
  • Keep tools reliable, tested, secure, and well documented.
Requirements
  • Professional experience building production software in TypeScript or Python.
  • Experience with data pipelines, APIs, and relational databases such as PostgreSQL.
  • Familiarity with language model APIs and practical constraints such as rate limits, cost, and nondeterministic outputs.
  • Care for correctness, testing, and reproducibility.
Benefits and work setup
  • Full-time role based in Scottsdale, Arizona, or remote within the United States.
  • Annual pay range of $140,000–$220,000.
  • Health insurance and paid time off.
Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior Research Engineer, AI Evaluation
Senior Research Engineer, AI Evaluation

system • United States

Hybrid
USD 180,000 - 240,000
Equity
Health and wellness coverage
Paid time off
+1
Senior Software Engineer – AI Evaluation
Senior Software Engineer – AI Evaluation

Alignerr • United States

Remote
USD 83,000 - 165,000
Senior Backend Engineer, AI Evaluations
Senior Backend Engineer, AI Evaluations

Ellipsis Health • San Francisco (CA)

Hybrid
USD 160,000 - 210,000
401(k) matching
Health insurance
Flexible PTO
Senior Python Engineer – LLM Evaluation (US-based)
Senior Python Engineer – LLM Evaluation (US-based)

Turing • United States

Remote
USD 69,000 - 96,000
Member of Technical Staff, Evaluations
Member of Technical Staff, Evaluations

Bake AI Inc. • San Mateo (CA)

On-site
USD 180,000 - 240,000
AI Evaluations Engineer
AI Evaluations Engineer

Ellipsis Health • San Francisco (CA)

Hybrid
USD 160,000 - 210,000
401(k) matching
Health insurance
Flex time off
AI Evaluation Engineer
AI Evaluation Engineer

Bask Health • New York (NY)

On-site
USD 120,000 - 170,000
Software Engineer, Evaluation Platform / Infra
Software Engineer, Evaluation Platform / Infra

Bison • San Francisco (CA)

On-site
USD 300,000 - 350,000
Health, dental, vision benefits
Unlimited PTO
Parental leave
+1
AI Evaluation Platform Engineer (Remote US)
AI Evaluation Platform Engineer (Remote US)

Modulus • United States

Remote
USD 140,000 - 220,000
Health insurance
Paid time off
Senior Software Engineer
Senior Software Engineer

cloudbedsthirdpartyboard • United States

Remote
USD 150,000 - 200,000
Remote-first
Home-office stipend
Paid time off
+2