AI or ML Engineer - Python, QA Testing

Optum; Hyderabad; On-site; INR 1,800,000 - 3,200,000; Full time

Job summary

Optum is seeking an AI Quality Engineer to define and execute a comprehensive quality strategy for agentic AI systems, ensuring accuracy, reliability, security, and safety across development and production.

You will build automation frameworks for prompts, model outputs, retrieval, tool use, and multi-step workflows, and collaborate with engineering and product teams to convert failures into lasting improvements.

Qualifications

• Bachelor's degree in Computer Science or a related field, or equivalent practical experience.
• 5+ years of experience in software quality engineering or test automation, including 2+ years testing AI/ML, generative AI, or LLM-based systems.
• Hands-on experience evaluating LLM applications with test datasets, rubrics, groundedness or relevance checks, and a combination of automated and human review.
• Experience designing and executing manual and automated functional, API, integration, end-to-end, regression, and risk-based tests.
• Experience integrating automated quality checks into CI/CD pipelines and testing APIs, databases, and distributed applications.
• Experience using logs, traces, dashboards, alerts, and user feedback to investigate production quality issues.
• Solid Python skills and experience building or extending test automation frameworks, such as Pytest or Playwright.
• Working knowledge of QA methods, defect lifecycle management, Agile delivery, and the software development lifecycle.
• Understanding of RAG, prompt engineering, tool use, and multi-step agent workflows.
• Demonstrated solid analytical, problem-solving, communication, and collaboration skills.

Responsibilities

• Define and execute a risk-based test strategy covering functional, API, integration, end-to-end, regression, performance, reliability, safety, and security testing for AI agents
• Build and maintain automated evaluation and test frameworks for prompts, model outputs, retrieval, tool calls, agent handoffs, and multi-step workflows
• Perform manual exploratory and scenario-based testing of new capabilities, ambiguous requests, edge cases, and real-world failure modes
• Create and maintain representative test datasets, expected outcomes, and evaluation rubrics. Use deterministic assertions, LLM-as-judge, and human review where each is appropriate
• Run quality checks for every release and for material prompt, model, tool, or data-source change. Integrate repeatable tests into CI/CD pipelines as release gates
• Monitor production quality using measures such as task completion, answer accuracy, groundedness, hallucinations, tool-call success, latency, cost, and user feedback
• Detect drift and quality regressions, set alert thresholds, investigate failures, and expand test coverage based on production findings
• Evaluate risks including prompt injection, sensitive-data exposure, bias, and unauthorized or unsafe agent actions
• Publish quality dashboards and release-readiness reports. Partner with engineering, product, and SRE teams on defect triage, root-cause analysis, and corrective actions
• Comply with all applicable Company policies, procedures, and business directives, changes including those relating to work location, team assignments, work schedules, and flexible work arrangements

Skills

Python

Test automation

CI/CD integration

Agile delivery

Analytical skills

Education

Bachelor's degree in Computer Science or related field

Tools

Pytest

Playwright

Job description

Job Summary

We are looking for an AI Quality Engineer to help ensure our agentic AI systems are accurate, reliable, secure, and safe. You will define and execute a quality strategy spanning manual testing, test automation, release evaluation, and production monitoring. You will build an automation framework for agent responses, retrieval, tool use, and multi-step workflows, and work with engineering and product teams to turn failures into lasting improvements.

Responsibilities

• Define and execute a risk-based test strategy covering functional, API, integration, end-to-end, regression, performance, reliability, safety, and security testing for AI agents
• Build and maintain automated evaluation and test frameworks for prompts, model outputs, retrieval, tool calls, agent handoffs, and multi-step workflows
• Perform manual exploratory and scenario-based testing of new capabilities, ambiguous requests, edge cases, and real-world failure modes
• Create and maintain representative test datasets, expected outcomes, and evaluation rubrics. Use deterministic assertions, LLM-as-judge, and human review where each is appropriate
• Run quality checks for every release and for material prompt, model, tool, or data-source change. Integrate repeatable tests into CI/CD pipelines as release gates
• Monitor production quality using measures such as task completion, answer accuracy, groundedness, hallucinations, tool-call success, latency, cost, and user feedback
• Detect drift and quality regressions, set alert thresholds, investigate failures, and expand test coverage based on production findings
• Evaluate risks including prompt injection, sensitive-data exposure, bias, and unauthorized or unsafe agent actions
• Publish quality dashboards and release-readiness reports. Partner with engineering, product, and SRE teams on defect triage, root-cause analysis, and corrective actions
• Comply with all applicable Company policies, procedures, and business directives, changes including those relating to work location, team assignments, work schedules, and flexible work arrangements

Required Qualifications

• Bachelor's degree in Computer Science or a related field, or equivalent practical experience
• 5+ years of experience in software quality engineering or test automation, including 2+ years testing AI/ML, generative AI, or LLM-based systems
• Hands-on experience evaluating LLM applications with test datasets, rubrics, groundedness or relevance checks, and a combination of automated and human review
• Experience designing and executing manual and automated functional, API, integration, end-to-end, regression, and risk-based tests
• Experience integrating automated quality checks into CI/CD pipelines and testing APIs, databases, and distributed applications
• Experience using logs, traces, dashboards, alerts, and user feedback to investigate production quality issues
• Solid Python skills and experience building or extending test automation frameworks, such as Pytest or Playwright
• Working knowledge of QA methods, defect lifecycle management, Agile delivery, and the software development lifecycle
• Understanding of RAG, prompt engineering, tool use, and multi-step agent workflows
• Demonstrated solid analytical, problem-solving, communication, and collaboration skills

Preferred Qualifications

• Experience with evaluation or observability tools such as LangSmith, Langfuse, Arize, DeepEval, Ragas, Vertex AI evaluation, or OpenTelemetry
• Experience with GCP, Vertex AI, or Gemini; experience testing applications on AWS or Azure
• Experience with JavaScript/TypeScript or Java, Selenium, advanced Playwright framework development, or browser testing platforms such as BrowserStack or Sauce Labs
• Experience with adversarial testing, responsible AI, and the OWASP Top 10 for LLM Applications
• Experience with A/B testing or canary evaluation of prompt and model changes
• Experience testing conversational AI, multi-agent or MCP-enabled solutions, event-driven services, or API orchestration
• Experience using AI to assist with test generation, failure analysis, or quality reporting
• Experience with performance and load testing tools such as JMeter or LoadRunner, or operational tools such as Dynatrace or SonarQube
• Experience with cloud migration, data migration, or data reconciliation testing
• Healthcare or other regulated-industry experience, including knowledge of HIPAA, data privacy, and security requirements. Experience with claims, eligibility, prior authorization, provider, contact center, or digital health systems