Senior Software Engineer, AI Evaluation Infrastructure

remote work; United States; Remote; USD 120,000 - 180,000; Full time

Job summary

Turing builds infrastructure for evaluating frontier AI systems. This role owns features across the stack that support data models, APIs, background workers, and UI for AI evaluation at scale.

Design end-to-end features, improve evaluation workflows, and generalize integration of new models and benchmarks. You will maintain performance and review PRs to uphold engineering standards as usage grows.

Qualifications

• 5+ years building and operating production backend systems with ownership of at least one service.
• Strong Python experience, plus Django or similar framework, PostgreSQL, and a task queue such as Celery.
• Hands on experience in LLM evaluation beyond simple API calls; understanding benchmark task construction and result aggregation.

Responsibilities

• Design and ship end to end features including data model, API, background jobs, and UI when needed.
• Improve evaluation workflows for agents and language models, including sandboxed trials and scoring results.
• Enhance benchmark task quality checks with verifiers, rubrics, and judge-based scoring.
• Generalize integration of new models, agents, and benchmarks without per item special cases.
• Maintain system performance and reliability as usage grows through efficient queries and safe migrations.
• Review teammates' pull requests and uphold engineering guidelines and codebase consistency.

Skills

Python

Backend ownership

LLM evaluation

TypeScript

React

Django

PostgreSQL

Celery

Tools

Celery

PostgreSQL

Job description

About the role

Turing builds infrastructure for evaluating frontier AI systems. This role owns features across the stack that support data models, APIs, background workers, and user interfaces for AI evaluation at scale.

What you'll do

• Design and ship end to end features including data model, API, background jobs, and UI when needed.
• Improve evaluation workflows for agents and language models, including sandboxed trials and scoring results.
• Enhance benchmark task quality checks with verifiers, rubrics, and judge-based scoring.
• Generalize integration of new models, agents, and benchmarks without per item special cases.
• Maintain system performance and reliability as usage grows through efficient queries and safe migrations.
• Review teammates' pull requests and uphold engineering guidelines and codebase consistency.

What they're looking for

• Typically 5+ years building and operating production backend systems with ownership of at least one service.
• Strong Python experience, plus Django or similar framework, PostgreSQL, and a task queue such as Celery.
• Hands on experience in LLM evaluation beyond simple API calls; understanding benchmark task construction and result aggregation.
• Comfort with a TypeScript and React frontend when changes are visible there.
• Ability to ship with minimal guidance, good judgment on tradeoffs, and clear written communication.
• Nice to have: experience with sandboxed environments, LLM as a judge, and distributed job scheduling at scale.

For applicants in Nigeria

• The listing states eligibility worldwide; no specific country restrictions beyond worldwide scope.
• Timezone expectations are not stated; confirm during interview.
• The listing does not specify contractor vs employee status; confirm during interview.