Senior AI Engineer - LLM Evals & Agents at ClickUp

Clickup; United States; Remote; USD 180,000 - 260,000; Full time

Job summary

ClickUp is hiring a Senior AI Engineer focused on LLM evaluations and agent harnesses. You will own how we measure and improve the quality of ClickUp's AI, building eval systems, core agent harnesses, and scalable backend services to support AI features across the platform.

You will design automated evaluation, manage model selection and routing across providers, and drive privacy and security practices while collaborating with product and data teams to deliver reliable, user-centric AI

Qualifications

• 5+ years of software engineering experience, including production LLM/ML systems.
• Hands-on experience building LLM evaluation systems in production.
• Strong experiment design and statistical rigor (A/B testing, offline/online metrics).
• Proven track record applying LLMs to real-world product features.
• Experience integrating multiple LLMs with benchmarking, routing, cost/latency tradeoffs.
• Backend engineering, distributed systems, and cloud-native technologies.

Responsibilities

• Design, build, and own ClickUp's AI evaluation system (golden datasets, LLM-as-judge, scorers).
• Develop and evolve the core agent harness and the AI that performs well on evals.
• Create self-improvement loops where agents modify others and prompts.
• Benchmark new models and own routing and fallback strategies across providers.
• Architect scalable AI platform services for deployment, orchestration, and lifecycle management of LLMs.
• Build robust APIs and backend systems to integrate AI features into ClickUp's platform.
• Develop infrastructure for model serving, monitoring, logging, and automated evaluation.

Skills

LLM evaluation systems

Backend engineering

Python

Distributed systems

Model routing

Tools

Kubernetes

Docker

AWS

GCP

Azure

Job description

Senior AI Engineer - LLM Evals & Agents at ClickUp — Remote

Market Positioning: Senior AI Engineer roles focused on LLM evaluation systems are among the hottest and most specialized positions in the AI industry right now. Compensation at this level typically ranges from $180,000–$260,000 total compensation at well-funded SaaS companies, with top-tier AI platforms offering equity-heavy packages. ClickUp's absence of a stated salary is a mild yellow flag — always negotiate with market data in hand.

Career Growth Signals: This role sits at the intersection of AI platform engineering and applied research, owning a critical function (AI quality measurement) that few engineers have deep experience in. Skills developed here — LLM eval design, agentic systems, model routing, and self-improvement loops — are highly transferable and in short supply, positioning you well for future Principal Engineer, AI Platform Lead, or Staff Researcher roles.

Green Flags:

• High ownership: you own the eval system end-to-end, not just contribute to it.
• AI-native company culture means you'll work with AI tooling daily, accelerating skill development.
• Exposure to multi-provider LLM strategy (OpenAI, Anthropic, Google) is rare and valuable resume depth.

Yellow Flags:

• No salary range disclosed — benchmark aggressively before accepting.
• ClickUp has had notable layoffs in past years; research current company trajectory and funding stage before committing.
• Broad scope (evals + agent harness + self-improvement loops + API infra) could indicate under-staffing in this area.

Work-Life Balance: Remote-first engineering culture is a positive signal, but "AI-native" and high ownership of mission-critical systems often correlates with demanding on-call and iteration cycles.

At ClickUp , we're building the future of work: the first truly converged AI workspace unifying tasks, docs, chat, calendar, and enterprise search, all supercharged by context-driven AI. We are an AI-native company. Every team member is expected to leverage AI daily, and we evaluate AI fluency as part of our hiring process. Join us and help redefine what's possible.

We're looking for a highly skilled and experienced Senior AI Engineer to join our ClickUp Engineering team, with a focus on AI evaluations. You'll own how we measure and improve the quality of ClickUp's AI: building the eval systems that tell us how well our AI is performing, the core agent harness, and the AI that performs well on those evals. You'll work on backend systems that make AI-powered capabilities scalable, reliable, and secure, while working hands‑on with LLMs to solve real user problems and drive product innovation.

Key Responsibilities:

• Design, build, and own ClickUp's AI evaluation system, including golden datasets, LLM-as-judge and automated scorers, regression detection, and quality tracking across AI features.
• Build and evolve the core agent harness, and the AI that performs well on those evals, partnering closely with product and engineering to define what "great" looks like.
• Build self-improvement loops where agents modify other agents or prompts, rerun evals, and iterate automatically.
• Benchmark new models as they're released and own model selection, routing, and fallback strategies across providers (e.g., OpenAI, Anthropic, Google, open-weight), balancing quality, latency, and cost.
• Architect, design, and implement scalable AI platform services that support the deployment, orchestration, and lifecycle management of LLMs and other AI models.
• Build and maintain robust APIs and backend systems that integrate AI-powered features into ClickUp's core platform.
• Develop infrastructure for model serving, monitoring, logging, and automated evaluation to ensure high reliability and performance of AI services in production.
• Drive best practices in AI privacy, security, and compliance, including data anonymization, secure data handling, and regulatory adherence.
• Collaborate cross-functionally with product, frontend, and data teams to deliver seamless, reliable, and user-centric AI experiences.

Qualifications:

• 5+ years of software engineering experience, including 2+ years building production LLM or ML systems.
• Hands‑on experience building LLM evaluation systems in production (golden datasets, LLM-as-judge, judge calibration, offline and online evals), tied to real product quality.
• Strong experiment design and statistical rigor, including A/B testing, offline vs. online metrics, and separating real regressions from noisy or variable results.
• Proven track record of applying LLMs and AI models to real‑world, customer‑facing product features.
• Proven experience integrating and managing multiple LLMs, including benchmarking, routing, and cost/latency/quality tradeoffs.
• Deep expertise in backend engineering, distributed systems, and cloud-native technologies (e.g., Kubernetes, Docker, AWS/GCP/Azure).
• Strong programming skills in Python (preferred), Go, TypeScript, or similar languages used for backend and AI platform development.
• Proficiency in orchestration frameworks and workflow engines (e.g., LangGraph, Airflow, Ray, or similar).
• Familiarity with search and retrieval technologies and their integration into AI-driven applications.
• Excellent collaboration and communication skills, with a track record of working effectively in cross‑functional teams.
• Passion for staying at the forefront of AI and applying new technologies to solve real‑world problems at scale.

Nice to Have:

• Background in data science, applied science, or machine learning engineering.
• Experience with eval tooling (e.g., Braintrust, LangSmith, Weave, Arize, or in‑house frameworks).
• Experience with model routing, AI gateways, or inference optimization.

Equal Opportunity Employer

ClickUp is an Equal Opportunity Employer, and qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender identity, or national origin.

Privacy Notice

ClickUp collects and processes personal data in accordance with applicable data protection laws. You can find further details by viewing our Global Candidate Privacy Notice.

Please note we are unable to sponsor or take over sponsorship of an employment visa for roles outside of engineering and product at this time. Sponsorship for engineering and product roles is not guaranteed, but is instead based on the business needs for that specific role at that time. Please reach out to the recruiter with any questions.

ClickUp Talent Acquisition will only initiate contact via an @ clickup.com email or through our official careers portal on clickup.com. We will never request fees, payments, or sensitive personal information. Please disregard any offers received outside these channels and report them to [e-mail].

AI Processing Notice

ClickUp may use artificial intelligence and machine learning technologies to help review and screen candidates' employment applications against role-related criteria. These tools support, but do not replace, human decision-making. If you have questions or need an accommodation in the recruitment process, please contact us at [e-mail].

Is this Senior AI Engineer role fully remote? Yes, the job posting is tagged #LI-REMOTE, confirming it is a fully remote position. Based on the location hint, it targets candidates based in the United States. Does ClickUp offer visa sponsorship for this engineering role? Visa sponsorship for engineering and product roles is possible but not guaranteed — it depends on the specific business needs for the role at the time. ClickUp recommends reaching out to the recruiter with any sponsorship questions. What LLM or AI experience is required for this role? The role requires 5+ years of software engineering experience, with at least 2+ years building production LLM or ML systems. Hands‑on experience with LLM evaluation systems (e.g., golden datasets, LLM-as-judge, judge calibration) is specifically required. What programming languages does ClickUp prefer for this AI Engineer role? Python is the preferred language, though Go and TypeScript are also listed as acceptable. Proficiency in orchestration frameworks such as LangGraph, Airflow, or Ray is also expected. What AI evaluation tools are considered a bonus for this position? Experience with eval tooling platforms such as Braintrust, LangSmith, Weave, or Arize is listed as a "nice to have," along with experience in model routing, AI gateways, or inference optimization.

Remote work & career advice that actually works.