AI Platform Engineer — Scalable Eval & LLM Benchmarks

Plexus Resource Solutions Ltd; United States; Remote; USD 140,000 - 210,000; Full time

Job summary

Plexus Resource Solutions Ltd. is partnering with an early-stage AI company to build an evaluation platform that benchmarks frontier models and agents for enterprise validation.

The role involves hands-on work with real ownership, collaborating directly with the founders. You will design and implement scalable evaluation pipelines, enable on-demand custom evals from client data, and ensure secure, multi-tenant services.

Qualifications

• Proven production experience building AI systems and infrastructure.
• Strong Python and backend or distributed-systems skills.
• Hands-on experience with LLM APIs and self-hosted models.
• Cloud platforms (AWS, GCP or Azure), Docker/Kubernetes and IaC.
• Thrives in an early-stage, fast-moving environment.

Responsibilities

• Build scalable pipelines for benchmark execution and results aggregation across multiple LLM providers and custom endpoints
• Develop systems that generate custom evals on demand from customer data
• Design secure, multi-tenant services and APIs to enterprise privacy standards
• Own orchestration, observability, performance and cost as usage grows

Skills

Python

Backend systems

Distributed systems

LLM APIs

Cloud

Docker/Kubernetes

Infrastructure as code

Early-stage environment

Tools

Docker

Kubernetes

Terraform

Job description

Plexus Resource Solutions Ltd. is partnering with an early-stage AI company to build an evaluation platform that benchmarks frontier models and agents for enterprise validation.

The role involves hands-on work with real ownership, collaborating directly with the founders. You will design and implement scalable evaluation pipelines, enable on-demand custom evals from client data, and ensure secure, multi-tenant services.