Remote Senior AI Engineer - LLM Eval & Agents

Clickup; United States; Remote; USD 180.000 - 260.000; Vollzeit

Zusammenfassung

ClickUp is hiring a Senior AI Engineer focused on LLM evaluations and agent harnesses. You will own how we measure and improve the quality of ClickUp's AI, building eval systems, core agent harnesses, and scalable backend services to support AI features across the platform.

You will design automated evaluation, manage model selection and routing across providers, and drive privacy and security practices while collaborating with product and data teams to deliver reliable, user-centric AI

Qualifikationen

• 5+ years of software engineering experience, including production LLM/ML systems.
• Hands-on experience building LLM evaluation systems in production.
• Strong experiment design and statistical rigor (A/B testing, offline/online metrics).
• Proven track record applying LLMs to real-world product features.
• Experience integrating multiple LLMs with benchmarking, routing, cost/latency tradeoffs.
• Backend engineering, distributed systems, and cloud-native technologies.

Aufgaben

• Design, build, and own ClickUp's AI evaluation system (golden datasets, LLM-as-judge, scorers).
• Develop and evolve the core agent harness and the AI that performs well on evals.
• Create self-improvement loops where agents modify others and prompts.
• Benchmark new models and own routing and fallback strategies across providers.
• Architect scalable AI platform services for deployment, orchestration, and lifecycle management of LLMs.
• Build robust APIs and backend systems to integrate AI features into ClickUp's platform.
• Develop infrastructure for model serving, monitoring, logging, and automated evaluation.

Kenntnisse

LLM evaluation systems

Backend engineering

Python

Distributed systems

Model routing

Tools

Kubernetes

Docker

AWS

GCP

Azure

Jobbeschreibung

ClickUp is hiring a Senior AI Engineer focused on LLM evaluations and agent harnesses. You will own how we measure and improve the quality of ClickUp's AI, building eval systems, core agent harnesses, and scalable backend services to support AI features across the platform.

You will design automated evaluation, manage model selection and routing across providers, and drive privacy and security practices while collaborating with product and data teams to deliver reliable, user-centric AI