LLMOps Engineer: AI Quality & Observability

Sovos; Lisboa; Presencial; EUR 60 000 - 90 000; Tempo integral

Vantagens oferecidas por esta oferta de emprego

Bi-Weekly Meeting Free Days

Mentoring and career development

Globally recognized training

Comprehensive health benefits

Resumo da oferta

Sovos is hiring an AI Ops Engineer to own the end-to-end LLMOps lifecycle for a portfolio of Ask Sovi products. You will gate AI agents, design evaluation frameworks, and build golden datasets to ensure reliable, safe AI experiences across the product lineup.

The role emphasizes monitoring quality, drift, latency, and cost, with a focus on continuous improvement and strong collaboration with AI, Product, and Engineering teams. UK shift adaptation noted.

Qualificações

• 1-3 years of experience in AI Engineering, AI Operations, AI Quality Engineering, LLMOps, MLOps, Test Automation, or related disciplines.
• Experience with Langfuse, LangSmith, Phoenix, or similar LLM observability/evaluation platforms.
• Strong understanding of LLMOps, AI quality engineering, and evaluation frameworks.
• Experience with hallucination detection, model drift monitoring, prompt regression management, and AI safety evaluations.
• Hands-on experience building evaluation datasets, quality benchmarks, and feedback loops.
• Familiarity with NLP concepts such as embeddings, semantic similarity, classification, and retrieval.
• Strong product mindset and stakeholder management skills.

Responsabilidades

• Gate new AI agents, prompts, and models before production release.
• Design and run evaluation frameworks using deterministic testing and LLM-as-a-Judge methodologies.
• Build and maintain golden datasets, annotation pipelines, and human-review workflows.
• Monitor AI quality, hallucinations, drift, latency, cost, and safety metrics.
• Convert customer feedback and production learnings into measurable quality improvements.
• Drive AI adoption and quality standards across your assigned product portfolio.

Conhecimentos

LLMOps

AI Quality Engineering

Evaluation Frameworks

NLP concepts

Langfuse/LangSmith

Ferramentas

Phoenix

Descrição da oferta de emprego

Sovos is hiring an AI Ops Engineer to own the end-to-end LLMOps lifecycle for a portfolio of Ask Sovi products. You will gate AI agents, design evaluation frameworks, and build golden datasets to ensure reliable, safe AI experiences across the product lineup.

The role emphasizes monitoring quality, drift, latency, and cost, with a focus on continuous improvement and strong collaboration with AI, Product, and Engineering teams. UK shift adaptation noted.