Senior ML Evaluations Platform Engineer - Remote

Artificial.Agency; United States; Hybrid; USD 85,000 - 127,000; Full time

Job summary

Artificial Agency is seeking a Senior Software Engineer to own the end-to-end ML evaluation platform used by researchers and engineers. You will manage the Python CLI, queueing service, API, database schema, and researcher-facing web UI, ensuring scalable, observable runs across Linux and GPU Windows workers.

The role emphasizes distributed systems, cloud infrastructure, and hands-on ML engineering collaboration with ML, agents, and game engineering teams.

Qualifications

• Bachelor's degree in Computer Science, Software Engineering, or related field, or equivalent practical experience.
• 5+ years of professional backend development experience, with a strong focus on Python.
• Experience building distributed job or workflow systems: queueing, scheduling, worker coordination, concurrency control, cancellation, and failure recovery.
• Strong relational database design (e.g., PostgreSQL) and schema migration experience.
• Familiarity with AWS, particularly EC2, IAM, Secrets Manager, and S3.
• Windows systems administration and PowerShell scripting.
• Experience designing CLIs, libraries, or internal tools for other engineers.
• Proficiency with GitHub workflows and modern CI/CD, including publishing internal packages.
• Experience with Docker and Kubernetes.
• Experience with ML experiment tracking or evaluation harnesses is a plus.
• Experience automating game engines (Unity, Unreal) in headless/batch modes is a plus.

Responsibilities

• Own our ML evaluation platform end to end: the Python CLI and library, queueing service and API, database schema, and the web UI researchers use to manage runs.
• Set technical direction and decide what the platform should and shouldn’t do.
• Design and operate the job execution layer: queueing, worker leasing and recovery, concurrency limits, cancellation, timeouts, retries, artifacts, and logs. Keep runs of hundreds of jobs correct, observable, and on time.
• Make evaluation results traceable and trustworthy. Record the relevant test versions, game builds, models, agent configurations, and run settings; preserve execution history; and distinguish valid evaluation outcomes from infrastructure failures, invalid tests, and incomplete runs.
• Extend the data model, APIs, and web UI to support baseline comparisons, regression investigation, and access to the logs and artifacts behind reported results. Work with ML, agents, and test engineering to implement agreed evaluation and reporting methods, including repeated trials and result aggregation.
• Working with our infrastructure team, own the machines the jobs run on: an auto-scaling GPU Windows fleet in AWS, containerized Linux workers on Kubernetes, and a few fixed in-office machines. This covers Terraform, machine images, GPU drivers, bootstrap scripts, secrets handling, and cost/capacity decisions.
• Partner with our Senior Test Engineer and other evaluation contributors on clear interfaces for workload configuration, execution requirements, structured results, and diagnostic artifacts. Make evaluations straightforward to run through developer tools and CI workflows.
• Work directly with ML, agents, game engineering, and research teams. Diagnose failed and stalled runs, unblock people quickly, and turn one-off requests into features that serve everyone.
• Prioritize across competing teams and deadlines, delivering on stakeholder requests while ensuring long-term platform health. Work with the Head of Platform Engineering to resolve conflicting business priorities.
• Keep runbooks accurate and actionable for other engineers to respond to incidents or outages. Maintain documentation and examples that help teams launch evaluations and integrate new workloads without routine manual intervention.

Skills

Python

Distributed systems

PostgreSQL

AWS

Docker

Kubernetes

CI/CD

GitHub Actions

Windows

CLI tooling

ML eval

Education

Bachelor's degree in CS

Tools

Terraform

Job description

Artificial Agency is seeking a Senior Software Engineer to own the end-to-end ML evaluation platform used by researchers and engineers. You will manage the Python CLI, queueing service, API, database schema, and researcher-facing web UI, ensuring scalable, observable runs across Linux and GPU Windows workers.

The role emphasizes distributed systems, cloud infrastructure, and hands-on ML engineering collaboration with ML, agents, and game engineering teams.