LLM Ops Lead

Sovos Latinoamérica; Lisboa; Presencial; EUR 45 000 - 75 000; Tempo integral

Vantagens oferecidas por esta oferta de emprego

Bi-Weekly Meeting Free Days

Mentoring and career development

Globally recognized training opportuni

Health and wellness benefits

Resumo da oferta

Sovos in Lisbon, Portugal, seeks an LLM Ops Lead with 2-3 years of hands-on experience to scale Generative AI across the business, ensuring quality, safety, observability, and continuous improvement. You will own AI quality standards, evaluation frameworks, and release gates for our AI-powered products.

You will lead prompt and model versioning, governance initiatives, safety programs, and AI incident response, coordinating with Product and Engineering.

Qualificações

• 2-3 years of experience in Applied AI, LLMOps, AI Quality Engineering, AI Operations, Test Automation, or related fields.
• Hands-on experience with Langfuse, LangSmith, or similar AI observability and evaluation tools.
• Strong understanding of AI quality, evaluation pipelines, prompt versioning, and AI governance.
• Experience with AI safety, guardrails, content moderation, audit logging, and release management.
• Strong stakeholder management and communication skills.
• Ability to support teams operating across UK business hours.

Responsabilidades

• Own AI quality standards, evaluation frameworks, and release gates across Ask Sovi products
• Lead prompt and model versioning, regression detection, and AI governance initiatives
• Coordinate Safety, Guardrails, and Red Team testing programs
• Build observability practices across latency, cost, quality, and safety metrics using tools such as Langfuse
• Manage Jira epics, ticket triage, and cross-functional coordination across Product and Engineering teams
• Develop LLMOps playbooks, best practices, and operational standards
• Lead AI incident response, troubleshooting, and root-cause analysis

Conhecimentos

AI governance

Stakeholder management

Communication

AI safety

Ferramentas

Langfuse

LangSmith

Descrição da oferta de emprego

Build your future with Sovos.

At Sovos, we're transforming tax compliance through innovative technology and AI-powered solutions used by customers worldwide.

We're looking for an LLM Ops Lead with 2-3 years of experience to help scale, govern, and operationalize Generative AI across the business while ensuring quality, safety, observability, and continuous improvement.

What You'll Do

• Own AI quality standards, evaluation frameworks, and release gates across Ask Sovi products
• Lead prompt and model versioning, regression detection, and AI governance initiatives
• Coordinate Safety, Guardrails, and Red Team testing programs
• Build observability practices across latency, cost, quality, and safety metrics using tools such as Langfuse
• Manage Jira epics, ticket triage, and cross-functional coordination across Product and Engineering teams
• Develop LLMOps playbooks, best practices, and operational standards
• Lead AI incident response, troubleshooting, and root-cause analysis

What We're Looking For

• 2-3 years of experience in Applied AI, LLMOps, AI Quality Engineering, AI Operations, Test Automation, or related fields
• Hands-on experience with Langfuse, LangSmith, or similar AI observability and evaluation tools
• Strong understanding of AI quality, evaluation pipelines, prompt versioning, and AI governance
• Experience with AI safety, guardrails, content moderation, audit logging, and release management
• Strong stakeholder management and communication skills
• Ability to support teams operating across UK business hours

Ideal Candidate Backgrounds

• Applied AI Engineers from AI-native or AI-first organizations with hands-on Langfuse/LangSmith experience
• Product-minded Engineers who combine strong technical skills with customer and business impact awareness
• QA/Test Automation Engineers who have transitioned into AI evaluation, quality, or LLM testing
• AI Quality Analysts with expertise in hallucination detection, model drift monitoring, LLM-as-a-Judge methodologies, and AI quality governance

Why Join Sovos?

• Work on cutting-edge Generative AI products and platforms
• Collaborate with global AI, Product, and Engineering leaders
• Bi-Weekly Meeting Free Days
• Mentoring and career development programs
Globally recognized training opportunities
• Comprehensive health and wellness benefits
• Flexible and collaborative work environment

Recruitment Process

CCAT & EPP Assessment → Automated Video Interview → Hiring Manager Interview → Final Interview

#Hiring #LLMOps #AIOps #GenerativeAI #AIEngineering #LangSmith #Langfuse #SovosCareers