Artificial Intelligence Engineer

Optum

Hyderabad

Híbrido

INR 1 500 000 - 2 100 000

Tempo integral

há 48 horas
Torna-te num dos primeiros candidatos

Recebe uma resposta deste empregador

O teu currículo e carta de apresentação, adaptados exatamente ao que estão a contratar.

Ultrapassa os filtros ATS

Resumo da oferta

Optum in Hyderabad seeks a seasoned QA and test automation engineer to lead risk-based testing for AI agents. You will build automated frameworks for prompts, model outputs, and multi-step workflows while performing manual exploratory testing of new capabilities.

You will design datasets, rubrics, and deterministic checks, and integrate tests into CI/CD. You will monitor metrics, detect drift, and collaborate with engineering, product, and SRE to ensure release quality and compliance.

Qualificações

  • Bachelor's degree in Computer Science or a related field.
  • 5+ years in software quality engineering or test automation, incl. 2+ years AI/ML testing.
  • Hands-on experience evaluating LLM applications with test datasets, rubrics, groundedness or relevance checks, and a combination of automated and human review.
  • Experience designing and executing manual and automated functional, API, integration, end-to-end, regression, and risk-based tests.
  • Experience integrating automated quality checks into CI/CD pipelines and testing APIs, databases, and distributed applications.

Responsabilidades

  • Define and execute a risk-based test strategy for AI agents.
  • Build and maintain automated evaluation frameworks for prompts and model outputs.
  • Perform manual exploratory testing of new capabilities and edge cases.
  • Create test datasets, outcomes, and evaluation rubrics with deterministic assertions.
  • Run quality checks for releases and integrate tests into CI/CD as gates.
  • Monitor production quality using metrics like accuracy, latency, and user feedback.
  • Detect drift, investigate failures, and expand test coverage.
  • Evaluate risks such as prompt injection, data exposure, bias, and unsafe actions.
  • Publish dashboards and coordinate with engineering, product, and SRE for defect triage.
  • Comply with company policies including work location and flexible arrangements.

Conhecimentos

QA engineering
Test automation
Python
Exploratory testing
Communication
Agile

Formação académica

Bachelor's degree in Computer Science or related field

Ferramentas

Pytest
Playwright
CI/CD
Logging/Monitoring

Descrição da oferta de emprego

Primary Responsibilities:
  • Define and execute a risk-based test strategy covering functional, API, integration, end-to-end, regression, performance, reliability, safety, and security testing for AI agents
  • Build and maintain automated evaluation and test frameworks for prompts, model outputs, retrieval, tool calls, agent handoffs, and multi-step workflows
  • Perform manual exploratory and scenario-based testing of new capabilities, ambiguous requests, edge cases, and real-world failure modes
  • Create and maintain representative test datasets, expected outcomes, and evaluation rubrics. Use deterministic assertions, LLM-as-judge, and human review where each is appropriate
  • Run quality checks for every release and for material prompt, model, tool, or data-source change. Integrate repeatable tests into CI/CD pipelines as release gates
  • Monitor production quality using measures such as task completion, answer accuracy, groundedness, hallucinations, tool-call success, latency, cost, and user feedback
  • Detect drift and quality regressions, set alert thresholds, investigate failures, and expand test coverage based on production findings
  • Evaluate risks including prompt injection, sensitive-data exposure, bias, and unauthorized or unsafe agent actions
  • Publish quality dashboards and release-readiness reports. Partner with engineering, product, and SRE teams on defect triage, root-cause analysis, and corrective actions
  • Comply with all applicable Company policies, procedures, and business directives, changes including those relating to work location, team assignments, work schedules, and flexible work arrangements
Qualifications - External
Required Qualifications:
  • Bachelor's degree in Computer Science or a related field
  • 5+ years of experience in software quality engineering or test automation, including 2+ years testing AI/ML, generative AI, or LLM-based systems
  • Hands-on experience evaluating LLM applications with test datasets, rubrics, groundedness or relevance checks, and a combination of automated and human review
  • Experience designing and executing manual and automated functional, API, integration, end-to-end, regression, and risk-based tests
  • Experience integrating automated quality checks into CI/CD pipelines and testing APIs, databases, and distributed applications
  • Experience using logs, traces, dashboards, alerts, and user feedback to investigate production quality issues
  • Solid Python skills and experience building or extending test automation frameworks, such as Pytest or Playwright
  • Working knowledge of QA methods, defect lifecycle management, Agile delivery, and the software development lifecycle
  • Understanding of RAG, prompt engineering, tool use, and multi-step agent workflows
  • Demonstrated solid analytical, problem-solving, communication, and collaboration skills
Preferred Qualifications:
  • Experience with evaluation or observability tools such as LangSmith, Langfuse, Arize, DeepEval, Ragas, Vertex AI evaluation, or OpenTelemetry
  • Experience with GCP, Vertex AI, or Gemini; experience testing applications on AWS or Azure
  • Experience with JavaScript/TypeScript or Java, Selenium, advanced Playwright framework development, or browser testing platforms such as BrowserStack or Sauce Labs
  • Experience with adversarial testing, responsible AI, and the OWASP Top 10 for LLM Applications
  • Experience with A/B testing or canary evaluation of prompt and model changes
  • Experience testing conversational AI, multi-agent or MCP-enabled solutions, event-driven services, or API orchestration
  • Experience using AI to assist with test generation, failure analysis, or quality reporting
  • Experience with performance and load testing tools such as JMeter or LoadRunner, or operational tools such as Dynatrace or SonarQube
  • Experience with cloud migration, data migration, or data reconciliation testing
  • Healthcare or other regulated-industry experience, including knowledge of HIPAA, data privacy, and security requirements. Experience with claims, eligibility, prior authorization, provider, contact center, or digital health systems
Obtém a tua avaliação gratuita e confidencial do currículo.

ou arrasta e larga o ficheiro aqui.

Similar jobs

Ofertas semelhantes que vale a pena comparar

AI or ML Engineer - Python, QA Testing
AI or ML Engineer - Python, QA Testing

Optum • Hyderabad

Presencial
INR 1 800 000 - 3 200 000
Senior QA Engineer, AI
Senior QA Engineer, AI

ICE • Hyderabad

Presencial
INR 1 800 000 - 3 000 000
AI Test Engineer
AI Test Engineer

AXA Global Business Services • Pune District, Bengaluru

Híbrido
INR 1 200 000 - 1 800 000
Senior QA Engineer, AI
Senior QA Engineer, AI

Intercontinental Exchange Holdings, Inc. • Hyderabad

Presencial
INR 2 000 000 - 4 000 000
Software Development Engineer in Test - AI
Software Development Engineer in Test - AI

Web.com Canada, Inc. • Índia

Presencial
INR 11 353 000 - 17 029 000
AI Test Engineer
AI Test Engineer

AXA Global Business Services (AXA GBS) • Pune District

Presencial
INR 900 000 - 1 500 000
AI Engineer
AI Engineer

Qentelli • Hyderabad

Presencial
INR 4 000 000 - 7 000 000
Senior QA AI Testing Engineer
Senior QA AI Testing Engineer

NewVision • Hyderabad

Presencial
INR 2 500 000 - 5 000 000
Lead AI Test Engineer
Lead AI Test Engineer

Qentelli LLC • Hyderabad

Presencial
INR 1 800 000 - 2 400 000
AI - Software Development Engineer in Test (AI - SDET)
AI - Software Development Engineer in Test (AI - SDET)

Mployee.me • Mumbai

Presencial
INR 1 200 000 - 1 800 000