Senior Product Engineer, Agent Training & Evals - Remote

Prolific Academic Ltd; United States; Remote; USD 120,000 - 180,000; Full time

Job summary

Prolific is seeking a Senior Product Engineer to join the Agent Training and Evals team. You’ll help build infrastructure to evaluate AI agents and capture agent traces at scale, working with real human expert tasks in secure environments.

You’ll shape the technical direction, collaborating with engineering, product, and AI research to turn ambiguous problems into scalable systems and see the impact of your work directly.

Qualifications

• Experience building scalable infrastructure or distributed systems in the cloud.
• Experience deploying, scaling, and securing systems.
• Experience evaluating AI models or agents.

Responsibilities

• Build tooling for capturing data from agents and humans performing real tasks at scale.
• Solve hard problems around compute, orchestration, scaling, security and reliability.
• Help develop approaches for training, benchmarking and evaluating AI agents.
• Work with emerging agentic AI technologies and rapidly evolving models.
• Make pragmatic technical decisions in a highly ambiguous environment.
• Partner with product engineering and AI research to turn customer problems into working products.
• Drive direct impact in a fast-moving, high-ownership, cross-functional team.

Skills

Cloud infrastructure

Distributed systems

Security best practices

AI model integration

Ambiguity tolerance

Python

Communication

Collaboration

Product mindset

Image/Video/Audio processing

Tools

FastAPI

Vue.js

PostgreSQL

GitHub Actions

Kubernetes

DataDog

Google Cloud Platform

Job description

Prolific is seeking a Senior Product Engineer to join the Agent Training and Evals team. You’ll help build infrastructure to evaluate AI agents and capture agent traces at scale, working with real human expert tasks in secure environments.

You’ll shape the technical direction, collaborating with engineering, product, and AI research to turn ambiguous problems into scalable systems and see the impact of your work directly.