Remote LLMOps Engineer — AI Quality & Observability

Sovos Latinoamérica; Lisboa; Presencial; EUR 40 000 - 70 000; Tempo integral

Vantagens oferecidas por esta oferta de emprego

Health and wellness benefits

Training opportunities

Resumo da oferta

Sovos is hiring an AI Ops Engineer to own the end-to-end LLMOps lifecycle for a portfolio of Ask Sovi products. You will drive AI quality, safety, evaluation, observability, and continual improvement to ensure reliable, scalable, and production-ready AI-powered experiences.

Responsibilities include gating agents, designing evaluation frameworks, and building golden datasets while collaborating across Product and Engineering teams.

Qualificações

• 1–3 years of experience in AI Engineering, AI Operations or related fields.
• Experience with LLM observability/evaluation platforms (Langfuse/LangSmith/Phoenix).
• Strong understanding of LLMOps, AI quality engineering and evaluation frameworks.
• Hands-on experience building evaluation datasets and quality benchmarks.
• Familiarity with NLP concepts: embeddings, semantic similarity, classification, retrieval.

Responsabilidades

• Gate new AI agents, prompts, and models before production release.
• Design and run evaluation frameworks with deterministic testing and LLM-as-a-Judge methods.
• Build and maintain golden datasets, annotation pipelines, and human-review workflows.
• Monitor AI quality, hallucinations, drift, latency, cost, and safety metrics.
• Translate customer feedback into measurable quality improvements.
• Drive AI adoption and quality standards across product portfolio.

Conhecimentos

LLMOps

MLOps

Test Automation

AI Quality Engineering

NLP concepts

Stakeholder management

Ferramentas

Langfuse

LangSmith

Phoenix

Descrição da oferta de emprego

Sovos is hiring an AI Ops Engineer to own the end-to-end LLMOps lifecycle for a portfolio of Ask Sovi products. You will drive AI quality, safety, evaluation, observability, and continual improvement to ensure reliable, scalable, and production-ready AI-powered experiences.

Responsibilities include gating agents, designing evaluation frameworks, and building golden datasets while collaborating across Product and Engineering teams.