Senior Software Engineer (Agent Training and Evals)

Prolific; United States; Remote; GBP 90,000 - 130,000; Full time

Benefits offered by this job

Remote work

Competitive salary and benefits

Job summary

Prolific is seeking a Senior Software Engineer to join the Agent Training & Evals team. You'll help build scalable infrastructure to evaluate AI agents at scale, turning research ideas into reliable production software and shaping the team's technical direction.

You’ll work with product, AI research and customers to turn problems into working products, own what you ship in production, and tackle data, compute, security and reliability challenges in a fast-moving, high-ownership environment.

Qualifications

• Background in Software Engineering, Data Engineering or MLOps.
• Experience building scalable infrastructure or distributed systems in the cloud.
• Solid understanding of deployment, scaling and security of systems.
• Hands-on experience integrating and evaluating AI models or agents.

Responsibilities

• Build a platform to support evaluating and fine-tuning agents.
• Take prototypes to production; turn experiments into reliable, well-tested software.
• Own what we ship as it scales in production.
• Solve hard problems around data, compute, orchestration, scaling, security and reliability.
• Work with emerging agentic AI technologies and rapidly evolving models.
• Make informed product and technical decisions in a highly experimental environment.
• Partner closely with customers, product engineering and AI research to turn problems into working products.
• Drive direct impact in a fast-moving, high-ownership, cross-functional team.

Skills

Software Engineering

Data Engineering

MLOps

Cloud infrastructure

Distributed systems

Python

Education

Bachelor's degree in Computer Science or related field

Tools

GitHub Actions

Kubernetes

Datadog

Langfuse

FastAPI

Vue.js

Postgres SQL

Job description

Senior Software Engineer - (Agent Training & Evals)

Prolific is not just another player in the AI space - we are the architects of the human data infrastructure that's reshaping the landscape of AI development. In a world where foundational AI technologies are increasingly commoditized, it's the quality and diversity of human-generated data that truly differentiates products and models.

As a Senior Software Engineer, you'll be one of the first engineers on our Agent Training & Evals team. You'll build infrastructure to evaluate AI agents at scale against real human expertise, in safe and secure environments. The better we get at this, the more useful, reliable and safe agents become. This is a genuine 0-1 role. There isn't a well-established playbook or existing architecture for you to simply maintain. You’ll shape the team’s technical direction and the products we build. You'll work closely with engineering, product and AI research to turn ambiguous problems into scalable systems, and you'll see the impact of what you build directly.

What you’ll be doing in the role

• Build a platform to support evaluating and fine-tuning agents
• Take prototypes to production; turn experiments and research ideas into reliable, well-tested software
• Own what we ship as it scales in production
• Solve hard problems around data, compute, orchestration, scaling, security and reliability
• Work with emerging agentic AI technologies and rapidly evolving models
• Make effective product and technical decisions in a highly experimental environment
• Partner closely with customers, product engineering and AI research to turn problems and opportunities into working products
• Drive direct impact in a fast-moving, high-ownership, cross-functional team

What you’ll bring to the role

• A background in Software Engineering, Data Engineering or MLOps
• Strong experience building scalable infrastructure or distributed systems in the cloud
• Solid understanding of how systems are deployed, scaled and secured
• Experience applying good engineering practice and raising the bar for those around you
• Hands-on experience with integrating and evaluating AI models or agents
• Interest in how AI agents are trained, evaluated and improved
• A product mindset with a focus on delivering business and customer value
• Comfort with ambiguity, and the ability to move quickly from idea to working system
• Python experience is useful but not essential

Key Technologies

• Cloud Platforms: Google Cloud Platform
• Programming Languages: Python, JavaScript, and TypeScript
• Frameworks: FastAPI, Vue.js, Container-based and Serverless architectures
• Databases: PostGres SQL
• DevOps and Monitoring: GitHub Actions, Kubernetes, Datadog, Langfuse

Why Prolific is a great place to work

We believe that the next leap in AI capabilities won't come solely from scaling existing models, but from integrating diverse human perspectives and behaviors into AI development. By providing this crucial human data infrastructure, Prolific is positioning itself at the forefront of the next wave of AI innovation - one that reflects the breadth and the best of humanity.

Working for us will place you at the forefront of AI innovation, providing access to our unique human data platform and opportunities for groundbreaking research.

Join us to enjoy a competitive salary, benefits, and remote working within our impactful, mission-driven culture.