Senior AI Backend Engineer — Agent Evaluation & Quality

Salla; Makkah Region; On-site; SAR 300,000 - 540,000; Full time

Job summary

Salla is seeking a senior engineer to own the evaluation stack for its production multi-agent systems. You will design LLM-as-judge components, calibrate against human labels, and quantify agent quality per failure mode.

You will also build simulators, integrate regression detection into CI, and contribute to agent development by turning failures into improvements. This role emphasizes strong software engineering and production readiness.

Qualifications

• 5+ years software engineering with recent hands-on LLM/agent work.
• Experience building evaluation stacks, judges, and simulators.
• Strong fundamentals in Python or TypeScript, APIs, testing and CI/CD.
• Hands-on experience with LangGraph, LangChain, or equivalent.

Responsibilities

• Own the evaluation stack and design LLM-as-judge systems.
• Integrate per-PR eval harnesses and regression detection into CI.
• Build user simulators to generate test coverage and adversarial cases.
• Turn real failures into improved evaluation data and criteria.
• Collaborate with product to formalize measurable success criteria.

Skills

LLM/agent experience

Measurement mindset

Production experience

System design

Tools

Python

TypeScript

LangGraph

LangChain

CI/CD

APIs

Job description

Salla is seeking a senior engineer to own the evaluation stack for its production multi-agent systems. You will design LLM-as-judge components, calibrate against human labels, and quantify agent quality per failure mode.

You will also build simulators, integrate regression detection into CI, and contribute to agent development by turning failures into improvements. This role emphasizes strong software engineering and production readiness.