Senior AI Engineer - Evaluation & LLM Platform (Remote)

Showcify, Inc.; United States; Remote; USD 180,000 - 260,000; Full time

Job summary

ClickUp is seeking a Senior AI Engineer to join the Engineering team with a focus on AI evaluations. You will own how we measure and improve the quality of ClickUp's AI, building eval systems, the core agent harness, and scalable backend services for LLMs.

You will collaborate with product and engineering to design experiments, benchmark models, and ensure robust, privacy-conscious AI features in production. This role is remote-friendly and impact-driven across core platform capabilities.

Qualifications

• 5+ years of software engineering experience, including production ML/LLM systems.
• Hands-on experience building LLM evaluation systems in production (golden datasets, judge calibration, offline/online evals).
• Strong experiment design and statistical rigor, including A/B testing and clear regression signals.
• Proven track record applying AI models to real product features.
• Experience with multiple LLMs, benchmarking, routing, and cost/latency/quality tradeoffs.

Responsibilities

• Design, build, and own ClickUp's AI evaluation system with golden datasets and automated scorers.
• Build and evolve the core agent harness and the AI that performs well on evals.
• Develop self-improvement loops where agents modify prompts and rerun evals automatically.
• Benchmark new models and own routing and fallback strategies across providers.
• Architect and implement scalable AI platform services for deployment and lifecycle management of LLMs.
• Build robust APIs and backend systems to integrate AI features into ClickUp.
• Develop infrastructure for model serving, monitoring, and logging for production reliability.
• Drive best practices in AI privacy, security, and regulatory adherence.
• Collaborate cross-functionally with product, frontend, and data teams.

Skills

LLM/ML systems

Backend engineering

Python

Go/TypeScript

Experiment design

Kubernetes

Docker

AWS

GCP

Azure

Tools

Kubernetes

Docker

AWS

GCP

Azure

Job description

ClickUp is seeking a Senior AI Engineer to join the Engineering team with a focus on AI evaluations. You will own how we measure and improve the quality of ClickUp's AI, building eval systems, the core agent harness, and scalable backend services for LLMs.

You will collaborate with product and engineering to design experiments, benchmark models, and ensure robust, privacy-conscious AI features in production. This role is remote-friendly and impact-driven across core platform capabilities.