LLM Evaluation Lead - Build Quality Metrics (Remote UK)

Ocho People Ltd.; Belfast City District; Remote; GBP 90,000 - 140,000; Full time

Benefits offered by this job

Equity

Remote work in UK

Job summary

Ocho People Ltd. is seeking a Machine Learning Engineer to build and own the evaluation function for production LLM and agent systems.

You’ll work across Python, NLP, and GCP, crafting benchmarks and automated scoring pipelines, with a hands-on role in shaping how AI quality is measured before release. The role is fully remote from Northern Ireland, offering a substantial base salary (£90,000–£140,000) plus equity, and will involve close collaboration with data science and engineering teams to

Qualifications

• 5+ years in ML engineering, NLP or applied data science.
• Hands-on experience with LLMs or agent-based systems.
• Strong Python and data-pipeline construction for evaluation datasets.
• Solid understanding of NLP and LLM capabilities including prompting techniques and retrieval.
• Strong statistics foundation including sampling and variance.
• Experience with evaluating mechanisms and benchmark systems for ML/LLM outputs.

Responsibilities

• Design evaluation frameworks and metrics for accuracy, safety, latency and cost.
• Build benchmark eval sets reflecting real customer scenarios and edge cases.
• Develop automated scoring pipelines using rubric-based grading and LLM-as-judge techniques.
• Calibrate automated judges against human review to ensure trustworthy numbers.
• Stand up regression suites to catch quality drops before release.
• Create dashboards tracking model and agent quality over time and releases.
• Investigate failure modes in multi-step agent behavior and prioritize fixes with engineering.
• Collaborate with senior data scientist on deeper statistical analyses.
• Balance evaluation coverage against compute and API cost.

Skills

ML engineering

NLP experience

Python pipelines

Statistical analysis

Education

Degree in CS/ML or related

Tools

GCP Vertex AI

BigQuery

Job description

Ocho People Ltd. is seeking a Machine Learning Engineer to build and own the evaluation function for production LLM and agent systems.

You’ll work across Python, NLP, and GCP, crafting benchmarks and automated scoring pipelines, with a hands-on role in shaping how AI quality is measured before release. The role is fully remote from Northern Ireland, offering a substantial base salary (£90,000–£140,000) plus equity, and will involve close collaboration with data science and engineering teams to