Senior ML Engineer - Evaluation & Agent Systems (Hybrid)

Ocho

Belfast City District

Hybrid

GBP 90,000 - 120,000

Full time

31 hours ago
Be an early applicant

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

Ocho, a Belfast-based AI-first software company, is seeking an ML Engineer to own evaluation end-to-end. You will design robust evaluation frameworks, automate scoring, and track model and agent behavior as the system scales.

The role combines engineering rigor with frontier LLM insights to deliver reliable, cost-efficient AI capabilities. You will collaborate with engineering and data science teams to build real-time dashboards, ensure high-confidence deployments, and push the envelope on

Qualifications

  • Bachelor's degree in Computer Science, Machine Learning, Statistics or a related field, or equivalent practical experience
  • 3+ years of experience in ML engineering, NLP or applied data science with hands-on exposure to LLM or agent-based systems
  • Deep practical experience building, chaining and evaluating autonomous agent workflows and frontier LLMs
  • Practical experience building or operating evaluation frameworks, automated scoring or benchmark systems for ML and LLM outputs
  • Strong Python skills and comfort building data pipelines for evaluation datasets
  • Solid understanding of NLP and modern LLM capabilities including prompting techniques, agentic workflows and retrieval
  • Working experience with Google Cloud Platform including Vertex AI and BigQuery, or equivalent AWS or Azure experience
  • Experience with inference optimisation and bridging frontier models into lower-cost internal systems
  • Experience with LLM-as-judge techniques, rubric design or human-in-the-loop evaluation programmes
  • Familiarity with agent architectures and the specific failure modes of multi-step and agentic systems
  • Experience operating evaluation systems at scale in a production environment

Responsibilities

  • Design evaluation frameworks and metrics covering accuracy, safety, latency and cost across agent and LLM systems
  • Build automated scoring pipelines, rubric-based grading and LLM-as-judge systems that scale beyond manual review
  • Design and build chained multi-agent architectures that underpin the core product capability
  • Stand up automated regression suites that catch quality drops from model, prompt or agent-logic changes before they reach production
  • Bridge high-cost frontier models into low-cost, optimised internal inference systems
  • Build dashboards and reporting that track model and agent quality over time and across releases
  • Identify failure modes and edge cases across diverse scenarios, working with the team to prioritise fixes
  • Partner closely with engineers building agent capabilities and with the senior data scientist for deeper analytical support

Skills

Python
NLP
LLM evaluation
Evaluation pipelines
Multi-agent systems
Data pipelines
GCP Vertex AI

Education

Bachelor's degree in Computer Science or related field

Tools

Vertex AI
BigQuery
AWS/Azure

Job description

Ocho, a Belfast-based AI-first software company, is seeking an ML Engineer to own evaluation end-to-end. You will design robust evaluation frameworks, automate scoring, and track model and agent behavior as the system scales.

The role combines engineering rigor with frontier LLM insights to deliver reliable, cost-efficient AI capabilities. You will collaborate with engineering and data science teams to build real-time dashboards, ensure high-confidence deployments, and push the envelope on

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Senior AI/ML Engineer: Production NLP & LLMs (Hybrid)
Senior AI/ML Engineer: Production NLP & LLMs (Hybrid)

Ocho People • Belfast City District

Hybrid
GBP 85,000 - 120,000
Hybrid work in Belfast
Senior Machine Learning Engineer
Senior Machine Learning Engineer

Ocho • Belfast City District

Hybrid
GBP 90,000 - 120,000
Founding ML Engineer, Evaluation & Agents — Ownership
Founding ML Engineer, Evaluation & Agents — Ownership

Tacit AI, Inc. • Greater London

On-site
GBP 90,000 - 130,000
Founding equity ownership
Access to production data from global
Ownership and impact on safety‑critial
Founding ML Engineer, Evaluation & Agents
Founding ML Engineer, Evaluation & Agents

Tacit AI, Inc. • Greater London

On-site
GBP 90,000 - 130,000
Founding equity ownership
Access to production data from global
Ownership and impact on safety‑critial
Senior AI Security Platform Engineer
Senior AI Security Platform Engineer

Ocho People Ltd. • Belfast City District

Hybrid
GBP 75,000 - 110,000
Senior ML Engineer - LLMs & Agentic AI (Hybrid Belfast)
Senior ML Engineer - LLMs & Agentic AI (Hybrid Belfast)

ViVA Tech Talent • Belfast

Hybrid
GBP 90,000 - 130,000
Bonus
Full Medical for you and your family
Very flexible working environment
Staff Software Engineer – AI Agent Security (Hybrid Belfast)
Staff Software Engineer – AI Agent Security (Hybrid Belfast)

Ocho People Ltd. • Belfast City District

Hybrid
GBP 90,000 - 130,000
Competitive salary plus bonus
Hybrid working from Belfast
Generous annual leave
+1
Principal AI Strategy & Enablement Engineer (Hybrid)
Principal AI Strategy & Enablement Engineer (Hybrid)

Ocho People • Belfast City District

Hybrid
GBP 85,000 - 130,000
Hybrid Senior AI Engineer: ML, Cloud & Impact
Hybrid Senior AI Engineer: ML, Cloud & Impact

Netcompany • Greater London

Hybrid
GBP 85,000 - 115,000
Hybrid working model
Private medical with Vitality
Pension contribution
+3
Senior ML QA Engineer: Benchmark & Validate AI Systems
Senior ML QA Engineer: Benchmark & Validate AI Systems

EngineersOfAI • Bristol

On-site
GBP 60,000 - 80,000
Unlimited annual leave
Up to 5% matched pension
Health cash plan
+2