Remote Senior AI Engineer, Evaluation Systems

Clickup; United States; Remote; USD 140,000 - 210,000; Full time

Job summary

ClickUp is hiring a Senior AI Engineer to own the AI evaluation stack, including golden datasets, LLM-as-judge, and automated scorers, while shaping the core agent harness and model routing. You will design self-improvement loops, benchmark models, and ensure scalable AI platform services across deployment and lifecycle management.

Ideal candidates combine backend engineering depth with hands-on ML system experience and strong collaboration across product and data teams, enabling reliable

Qualifications

• 5+ years of software engineering experience, including 2+ years building production LLM or ML systems.
• Hands-on experience building LLM evaluation systems in production (golden datasets, LLM-as-judge, judge calibration, offline and online evals).
• Strong experiment design and statistical rigor, including A/B testing, offline vs. online metrics, and separating real regressions from noise.
• Proven track record applying LLMs and AI models to real-world product features.
• Proven experience integrating and managing multiple LLMs, including benchmarking, routing, and cost/latency/quality tradeoffs.
• Deep expertise in backend engineering, distributed systems, and cloud-native tech (Kubernetes, Docker, AWS/GCP/Azure).
• Strong programming skills in Python (preferred), Go, TypeScript, or similar languages.
• Proficiency in orchestration frameworks and workflow engines (LangGraph, Airflow, Ray, or similar).
• Familiarity with search and retrieval technologies and their integration into AI-driven apps.
• Excellent collaboration and communication skills in cross-functional teams.
• Passion for staying at the forefront of AI and applying new technologies at scale.

Responsibilities

• Design, build, and own ClickUp's AI evaluation system, including golden datasets, LLM-as-judge and automated scorers.
• Build and evolve the core agent harness, and the AI that performs well on evals, aligning with product and engineering.
• Build self-improvement loops where agents modify other agents or prompts and rerun evals automatically.
• Benchmark new models and own model selection, routing, and fallback strategies across providers.
• Architect, design, and implement scalable AI platform services for deployment, orchestration, and lifecycle management of LLMs.
• Build and maintain robust APIs and backend systems that integrate AI-powered features into ClickUp's core platform.
• Develop infrastructure for model serving, monitoring, logging, and automated evaluation for reliable AI services.
• Drive best practices in AI privacy, security, and compliance, including data anonymization and secure handling.
• Collaborate cross-functionally with product, frontend, and data teams to deliver reliable AI experiences.

Skills

5+ years software eng

LLM evaluation systems

Experiment design

LLM applications

LLM orchestration

Backend engineering

Python/Go/TypeScript

Orchestration tools

Search tech

Collaboration

AI leadership

Tools

Kubernetes

Docker

AWS/GCP/Azure

Job description

ClickUp is hiring a Senior AI Engineer to own the AI evaluation stack, including golden datasets, LLM-as-judge, and automated scorers, while shaping the core agent harness and model routing. You will design self-improvement loops, benchmark models, and ensure scalable AI platform services across deployment and lifecycle management.

Ideal candidates combine backend engineering depth with hands-on ML system experience and strong collaboration across product and data teams, enabling reliable