AI Engineer

Plexus RS; United States; Remote; USD 140,000 - 210,000; Full time

Job summary

Plexus Resource Solutions Ltd. is partnering with an early-stage AI company to build an evaluation platform that benchmarks frontier models and agents for enterprise validation.

The role involves hands-on work with real ownership, collaborating directly with the founders. You will design and implement scalable evaluation pipelines, enable on-demand custom evals from client data, and ensure secure, multi-tenant services.

Qualifications

• Proven production experience building AI systems and infrastructure.
• Strong Python and backend or distributed-systems skills.
• Hands-on experience with LLM APIs and self-hosted models.
• Cloud platforms (AWS, GCP or Azure), Docker/Kubernetes and IaC.
• Thrives in an early-stage, fast-moving environment.

Responsibilities

• Build scalable pipelines for benchmark execution and results aggregation across multiple LLM providers and custom endpoints
• Develop systems that generate custom evals on demand from customer data
• Design secure, multi-tenant services and APIs to enterprise privacy standards
• Own orchestration, observability, performance and cost as usage grows

Skills

Python

Backend systems

Distributed systems

LLM APIs

Cloud

Docker/Kubernetes

Infrastructure as code

Early-stage environment

Tools

Docker

Kubernetes

Terraform

Job description

Plexus has partnered with an early-stage AI company building an evaluation platform that helps enterprise teams validate generative AI models and agents before they go live. The product is already live, with a public leaderboard benchmarking leading LLMs, and the team is now growing.

The role

You will build the infrastructure that runs evaluations of frontier models and agents at scale. This is a hands-on role with real ownership, working directly with the founders.

Responsibilities

• Build scalable pipelines for benchmark execution, scoring and results aggregation across multiple LLM providers and custom endpoints
• Develop systems that generate custom evals on demand from customer data
• Design secure, multi-tenant services and APIs to enterprise privacy standards
• Own orchestration, observability, performance and cost as usage grows

Experience required

• Proven experience building AI systems and AI infrastructure in production
• Strong Python (or similar) and backend or distributed systems skills
• Hands-on experience with LLM APIs and open or self-hosted models
• Cloud (AWS, GCP or Azure), Docker/Kubernetes and infrastructure-as-code
• Comfortable in an early-stage, fast-moving environment

Nice to have

• Experience with LLM evaluation, benchmarking or agent frameworks
• Startup or scale-up background

Comp: Base + Equity