Senior AI Engineer - Production LLMs & Evaluation

Learndevs; United States; Remote; USD 200.000 - 250.000; Vollzeit

Zusammenfassung

ClickUp is seeking a Senior AI Engineer to own the AI evaluation systems that measure and improve ClickUp’s AI quality. You will build the core agent harness, model routing, and scalable backend services powering AI features across the platform.

You’ll work hands-on with LLMs, design experiments, and collaborate with product, frontend, and data teams to deliver reliable, user-centric AI capabilities at scale.

Qualifikationen

• 5+ years of software engineering experience, including 2+ years building production LLM or ML systems.
• Hands-on experience building LLM evaluation systems in production (golden datasets, LLM-as-judge, judge calibration, offline and online evals).
• Strong experiment design and statistical rigor, including A/B testing and metrics.
• Proven ability applying LLMs to real-world, customer-facing features.

Aufgaben

• Design, build, and own ClickUp’s AI evaluation system, including golden datasets, LLM-as-judge and automated scorers, regression detection, and quality tracking across AI features.
• Build and evolve the core agent harness, and the AI that performs well on those evals, partnering with product and engineering.
• Build self-improvement loops where agents modify other agents or prompts, rerun evals, and iterate automatically.
• Benchmark new models and own model routing, selection, and fallback strategies across providers.
• Architect, design, and implement scalable AI platform services for deployment, orchestration, and lifecycle management of LLMs.
• Build and maintain robust APIs and backend systems for AI-powered features.
• Develop infrastructure for model serving, monitoring, logging, and automated evaluation to ensure reliability.
• Drive AI privacy, security, and regulatory compliance, including data anonymization.

Kenntnisse

LLM evaluation systems

Distributed systems

Python

Go

TypeScript

Kubernetes

AWS/GCP/Azure

Tools

LangGraph

Airflow

Ray

Docker

Kubernetes

Jobbeschreibung

ClickUp is seeking a Senior AI Engineer to own the AI evaluation systems that measure and improve ClickUp’s AI quality. You will build the core agent harness, model routing, and scalable backend services powering AI features across the platform.

You’ll work hands-on with LLMs, design experiments, and collaborate with product, frontend, and data teams to deliver reliable, user-centric AI capabilities at scale.