Senior Software Engineer: AI Agent Training & Evaluations

AI Trainer Jobs; United States; Remote; USD 150,000 - 190,000; Full time

Benefits offered by this job

Remote working

Competitive salary

Job summary

Prolific is seeking a Senior Software Engineer to join the Agent Training and Evals team. You will build scalable infrastructure to evaluate AI agents with real user-style data, collaborating with product, AI research, and engineering.

You’ll be among the first engineers in this team, shaping technical direction and product strategy while delivering reliable software for production environments and high-impact experiments.

Qualifications

• Background in Software Engineering, Data Engineering or MLOps.
• Experience building scalable infrastructure or distributed systems in the cloud.
• Hands-on experience with integrating and evaluating AI models or agents.

Responsibilities

• Build a platform to support evaluating and fine-tuning agents.
• Take prototypes to production; turn experiments and research ideas into reliable, well-tested software.
• Own what we ship as it scales in production.
• Solve hard problems around data, compute, orchestration, scaling, security and reliability.
• Work with emerging agentic AI technologies and rapidly evolving models.
• Make effective product and technical decisions in a highly experimental environment.
• Partner closely with customers, product engineering and AI research to turn problems and opportunities into working products.
• Drive direct impact in a fast-moving, high-ownership, cross-functional team.

Skills

Software Engineering

Data Engineering

MLOps

Cloud Experience

Python

Tools

Google Cloud Platform

FastAPI

Vue.js

PostgreSQL

Kubernetes

Datadog

Langfuse

GitHub Actions

Job description

Prolific is seeking a Senior Software Engineer to join the Agent Training and Evals team. You will build scalable infrastructure to evaluate AI agents with real user-style data, collaborating with product, AI research, and engineering.

You’ll be among the first engineers in this team, shaping technical direction and product strategy while delivering reliable software for production environments and high-impact experiments.