AI Ops Engineer

Sovos; Navi Mumbai; On-site; INR 800,000 - 1,200,000; Full time

Benefits offered by this job

Remote-first collaboration

Bi-Weekly Meeting Free Days

Mentoring and career development

Job summary

Sovos is seeking an AI Ops Engineer to own the end-to-end LLMOps lifecycle for a portfolio of Ask Sovi products. You will drive AI quality, safety, evaluation, observability, and continuous improvement to ensure AI-powered experiences are reliable and production-ready.

Ideal candidates have 1–3 years in AI Engineering or related fields, with hands-on experience in Langfuse/LangSmith/Phoenix and a strong product mindset for stakeholder management.

Qualifications

• 1–3 years of experience in AI Engineering or related fields.
• Experience with LLM observability/evaluation platforms (Langfuse, LangSmith, Phoenix).
• Strong understanding of LLMOps, AI quality engineering, and evaluation frameworks.
• Experience with hallucination detection, drift monitoring, prompt regression management, and AI safety evaluations.
• Hands-on experience building evaluation datasets and feedback loops.
• Familiarity with NLP concepts like embeddings and retrieval.

Responsibilities

• Gate new AI agents, prompts, and models before production release.
• Design and run evaluation frameworks using deterministic testing.
• Build and maintain golden datasets, annotation pipelines, and human-review workflows.
• Monitor AI quality, hallucinations, drift, latency, cost, and safety metrics.
• Convert customer feedback into measurable quality improvements.
• Drive AI adoption and quality standards across product portfolios.

Skills

LLMOps

LangSmith

MLOps

Evaluation Frameworks

Tools

Langfuse

Phoenix

Job description

Build your future with Sovos.

Sovos is transforming tax compliance from a business requirement into a force for growth. Trusted by more than 100,000 customers worldwide, including many of the world's leading organizations, we are leveraging AI to build smarter, more impactful customer experiences.

We're looking for a AI Ops Engineer to own the end-to-end LLMOps lifecycle for a portfolio of Ask Sovi products. You'll drive AI quality, safety, evaluation, observability, and continuous improvement, ensuring our AI-powered experiences are reliable, scalable, and production-ready.

What You'll Do

• Gate new AI agents, prompts, and models before production release
• Design and run evaluation frameworks using deterministic testing and LLM-as-a-Judge methodologies
• Build and maintain golden datasets, annotation pipelines, and human-review workflows
• Monitor AI quality, hallucinations, drift, latency, cost, and safety metrics
• Convert customer feedback and production learnings into measurable quality improvements
• Drive AI adoption and quality standards across your assigned product portfolio

What We're Looking For

• 1-3 years of experience in AI Engineering, AI Operations, AI Quality Engineering, LLMOps, MLOps, Test Automation, or related disciplines
• Experience with Langfuse, LangSmith, Phoenix, or similar LLM observability/evaluation platforms
• Strong understanding of LLMOps, AI quality engineering, and evaluation frameworks
• Experience with hallucination detection, model drift monitoring, prompt regression management, and AI safety evaluations
• Hands‑on experience building evaluation datasets, quality benchmarks, and feedback loops
• Familiarity with NLP concepts such as embeddings, semantic similarity, classification, and retrieval
• Strong product mindset and stakeholder management skills

Ideal Backgrounds

• Applied AI Engineers from AI-native or AI-first organizations
• Product‑minded Engineers with strong customer and business awareness
• QA/Test Automation Engineers who have transitioned into AI evaluation
• AI Quality Analysts with expertise in hallucination detection, drift monitoring, and LLM‑as‑a‑Judge frameworks

Working Hours

UK Shift

Why Join Sovos?

• Work on cutting-edge Generative AI products
• Global, remote‑first collaboration with AI, Product, and Engineering leaders
• Bi‑Weekly Meeting Free Days
• Mentoring and career development programs
• Globally recognized training opportunities
• Comprehensive health and wellness benefits
• Opportunity to shape AI experiences used by customers around the world

Recruitment Process

1. CCAT & EPP Assessment
2. Automated Video Interview
3. Hiring Manager / Team Interview
4. Final Interview

Interested in building the future of Enterprise AI with Sovos? We'd love to hear from you.