AI Quality Automation Engineer: Python & AI Testing

Luxoft Germany; United States; Remote; USD 90,000 - 120,000; Full time

Job summary

Luxoft is seeking an experienced software quality engineer to lead automated testing for AI-driven systems. You will build reusable test frameworks for agents, prompts, and workflows, and develop benchmarks, regression suites, and release thresholds.

Responsibilities include evaluating hallucination, grounding, citation accuracy, and prompt injection; validating API/UI/integration/CI/CD tests; and monitoring production quality, drift, and recurring failures.

Qualifications

• Bachelor's degree in Computer Science, Software Engineering, Information Systems, or a related field.
• 5+ years of software quality engineering, test automation, or reliability engineering experience.
• Strong Python skills and experience building reusable automated test frameworks.
• Experience with API, UI, integration, regression, and CI/CD-based testing, including measurable quality gates.
• Ability to translate requirements, business workflows, and AI behaviors into measurable quality criteria.

Responsibilities

• Build automated evaluation frameworks for agents, RAG, prompts, tools, workflows, and model outputs. Create benchmark datasets, golden responses, regression suites, evaluation metrics, and release thresholds.
• Test hallucination, grounding, citation accuracy, prompt injection, tool selection, and multi-agent coordination.
• Develop agents that generate test cases, validate API and workflow outputs, and support intelligent UAT.
• Build UI, API, integration, performance, end-to-end automation, and CI/CD quality gates.
• Monitor production quality, drift, reliability, and recurring failure patterns.

Job description

Luxoft is seeking an experienced software quality engineer to lead automated testing for AI-driven systems. You will build reusable test frameworks for agents, prompts, and workflows, and develop benchmarks, regression suites, and release thresholds.

Responsibilities include evaluating hallucination, grounding, citation accuracy, and prompt injection; validating API/UI/integration/CI/CD tests; and monitoring production quality, drift, and recurring failures.