Senior AI Engineer - Evaluation Systems (Remote)

Devconnectplatform; United States; Remote; USD 200,000 - 250,000; Full time

Job summary

ClickUp is seeking a Senior AI Developer to lead the design and implementation of scalable AI evaluation infrastructure. You will own the evaluation datasets, model routing, and performance metrics used to drive product quality across AI features.

You will work on backend systems, integrate multiple LLM providers, and build robust APIs for AI-powered capabilities. Collaboration with product, data, and engineering teams is essential to deliver reliable, privacy-conscious solutions at scale.

Qualifications

• Experience designing and building production LLM or ML systems.
• Strong experimental design, A/B testing, and metric interpretation.
• Proven track record applying LLMs to customer-facing features.
• Experience integrating multiple LLMs with routing and cost/latency tradeoffs.
• Backend engineering with cloud-native technologies.

Responsibilities

• Design, build, and own AI evaluation systems (datasets, judge, scorers).
• Develop and maintain core agent harnesses and AI components.
• Benchmark models and manage model selection, routing, and fallbacks.
• Build scalable AI platform services and robust APIs for AI features.
• Ensure data privacy, security, and regulatory compliance.

Skills

LLM evaluation

Backend systems

Distributed systems

Python

Go

Cloud knowledge

Experiment design

Education

Bachelor's degree in CS/engineering

Tools

Kubernetes

Docker

AWS

GCP

Azure

Job description

ClickUp is seeking a Senior AI Developer to lead the design and implementation of scalable AI evaluation infrastructure. You will own the evaluation datasets, model routing, and performance metrics used to drive product quality across AI features.

You will work on backend systems, integrate multiple LLM providers, and build robust APIs for AI-powered capabilities. Collaboration with product, data, and engineering teams is essential to deliver reliable, privacy-conscious solutions at scale.