AI engineer in secure enclaves

Valarian; Greater London; Hybrid; GBP 90,000 - 120,000; Full time

Benefits offered by this job

Акционерное участие

Конкурентная зарплата

Взносы в пенсионный фонд

Частная медицинская страховка

Гибридный формат работы

Корпоративные выезды/ретриты

Job summary

Valarian Technologies привлекает инженера-исследователя для разработки и сопровождения масштабируемых Python‑оценочных рангов, измеряющих выполнение задач, качество траекторий, использование инструментов и стоимость. Ваша работа включает применение статистических методов и разработку автоматических judging‑систем с точки зрения человеческого эталона.

Ключевые задачи включают анализ согласованности между аннотаторами, обнаружение ошибок и предложение архитектурных улучшений в контурах агентов,

Qualifications

• Знание статистических методов: гипотезы, бутстрэп, анализ мощности.
• Умение настраивать и валидировать параметры оценки существующих систем.
• Опыт в создании наборов данных для оценки и формулировании рубрик оценки.

Responsibilities

• Разработка и поддержка масштабируемых Python‑оценочных рангов, измеряющих качество выполнения задач и использование инструментов.
• Применение статистических методов для различения улучшений системы и шума.
• Разработка автоматических judging‑систем, сопоставимых с человеческим эталоном.
• Оценка согласованности аннотаторов и анализ источников ошибки.
• Внедрение архитектурных улучшений в контурах агентов и дизайне подсистем.

Skills

Hypothesis testing

Bootstrapping

Power analysis

LLM judge setups

Inter‑annotator agreement

Evaluation datasets

Prompt engineering

Python

LangChain

Tools

Python

LangChain

lm-evaluation-harness

Promptfoo

Ragas

Kubernetes

Tool calling

Job description

Описание:

Valarian Technologies is a dual-use technology company building critical tools to safeguard the future in an era of evolving global security challenges. We're rethinking security beyond traditional military domains, addressing asymmetric threats that impact our technological advantage, economic strength, and democratic institutions. We build Acra – the platform foundation for everything we do as a dual-use technology company.

Задачи:

Design and maintain scalable Python evaluation harnesses that measure task completion, trajectory quality, tool‑use correctness, cost, latency, and variance across multi-step agentic systems. Apply rigorous statistical methods—including hypothesis testing, confidence intervals, bootstrapping, and power analysis to determine whether performance changes are true system improvements or stochastic noise. Develop automated judging systems validated against human ground truth. Measure alignment using agreement metrics (Cohen's kappa, correlation, precision/recall) and systematically detect judge failure modes such as position bias, verbosity bias, self‑preference, and prompt sensitivity. Define sampling strategies, detailed annotation guidelines, and scoring rubrics. Measure inter‑annotator agreement and manage label noise to ensure benchmark integrity over time. Conduct hands‑on failure analysis on agent runs, cluster root causes into actionable error categories, and clearly communicate findings to both engineering teams and stakeholders. Translate evaluation results directly into architectural improvements across agent scaffolding, prompt design, tool execution loops, context window compaction, retries, and safety guardrails.

Требования:

Demonstrated expertise in hypothesis testing, bootstrapping, power analysis, and statistical significance within probabilistic machine learning systems. Proven experience building, auditing, and validating LLM judge setups against human annotations, including tracking agreement metrics and mitigating judge biases. Strong track record designing evaluation datasets, crafting precise scoring rubrics, managing label noise, and measuring inter‑annotator agreement. Deep comfort evaluating complex agent trajectories, tool execution sequences, latency, cost, and variance across repeated runs rather than relying solely on single‑shot benchmarks. Exceptional judgment in defining metrics tied to real‑world outcomes, with a keen eye for detecting benchmark gaming, target drift, or shortcut learning. Ability to dissect complex execution logs, categorize failure modes, and synthesize clear, data‑driven recommendations. Practical proficiency with LLM APIs, prompt engineering, structured outputs, function/tool calling, and retrieval systems in Python. Clear understanding of planning loops, tool orchestration, memory management, and multi‑agent coordination, including their trade‑offs and common failure modes. Ability to write clean, maintainable Python code to iterate on harness components such as guardrails, retry logic, state handling, and context management. Будет плюсом: Experience with agent and evaluation frameworks (e.g., LangChain, LangGraph, AutoGen, lm‑evaluation‑harness, Promptfoo, or Ragas); Exposure to running evaluation pipelines or agent workloads within secure, enclave, or Kubernetes environments

Условия:

• Equity – because you have the right to own what you’re building
• A competitive salary – because we value your unique skills
• Employer pension contributions – because you deserve a secure future
• Private health insurance - because your health is important to us
• Hybrid work setup – because everyone has different needs
• Rewarding company retreats and meetups that respect your work/life balance – because we love getting to know each other!

We operate on a hybrid model, gathering in our London office 3 days a week to support one another and collaborate.