Test Engineer – AI Voice Systems (w/m/d) - experience in test automation, Python, LLM. Azure Speech- Immediate Joiners

Luxoft Germany; Ingolstadt; On-site; EUR 90,000 - 120,000; Full time

Job summary

Luxoft Germany in Ingolstadt is seeking a senior QA/quality engineer to own the AI evaluation platform for in-vehicle assistants. You will design scoring, evaluate LLM outputs, and drive end-to-end test strategies across audio, dialog, and backend components.

You will work onsite in the Ingolstadt/Munich area, collaborating with backend, frontend, AI and hardware teams to gate releases with robust metrics and automated tests.

Qualifications

• 5+ years in test automation / quality engineering with strong Python.
• Hands-on experience evaluating LLM or other non‑deterministic systems.
• Practical command of LLM quality metrics (faithfulness, hallucination, relevance).
• Experience with LangChain / LangGraph and LLM observability tools.
• Prompt regression testing in practice.
• Solid conventional test engineering: pytest, code coverage, test design.
• Experience integrating ASR and TTS services (Azure Speech, Whisper, etc.).
• CI/CD integration with Azure DevOps/GitHub Actions and Docker; REST APIs & SQL.
• English C1; willing to work onsite in Ingolstadt/Munich area.

Responsibilities

• Build and own the AI evaluation engine: discrete and continuous scoring with confidence values.
• Define and maintain the quality metric set for an automotive assistant.
• Develop generative test cases and dialogue generation for domains like navigation, media, knowledge.
• Make generated dialogue realistically human with multi-turn conversations.
• Engineer test variability and reliability with randomised parameters.
• Build prompt and model regression suites in LangFuse; detect drift.
• Contribute to audio simulation and multi-speaker testing with realistic noise.
• Cover conventional test engineering: pytest, testcontainers, code coverage.
• Support test execution in real vehicles and on test racks; health checks and triage.
• Deliver reporting and analytics; export results and track performance trends.
• Maintain traceability between test cases, requirements and defects.
• Coordinate with backend, frontend, AI and hardware teams; report status to management.

Skills

Python

LLM evaluation

LLM quality metrics

LangChain / LangGraph

Prompt regression

Test engineering

ASR / TTS integration

CI/CD & Docker

English proficiency

Tools

pytest

Docker

REST APIs

SQL / PostgreSQL

Job description

Our client is advancing its in-vehicle voice assistant into an intelligent, AI-powered companion. Large-language-model capabilities (Azure OpenAI / ChatGPT) have been running in production across vehicles with new E³-architecture models featuring enhanced voice functions from the factory. The customer backend is the cloud AI orchestration service behind this: it receives requests from the vehicle, classifies and routes them, orchestrates the LLM, tool services and specialised agents, and returns an answer or action to the car. DXC Luxoft serves as the end-to-end delivery partner, working in a joint product team with the client’s engineers on the Azure platform (AKS, Azure OpenAI, AI Foundry, Managed Identity, Azure Monitor, Azure DevOps).

This role sits in the end-to-end testing and AI evaluation work package — the quality gate for the whole platform. LLM-based assistants generate variable, context-dependent answers: every response is unique and only conditionally predictable, so classical test systems with rigid expected values cannot cover them. The work package therefore builds and operates an intelligent, AI-supported E2E test platform that reacts dynamically to each assistant output and continues the test dialogue appropriately.

Concretely, the platform generates synthetic speech with variable parameters (languages, pitch, emotional colouring, natural variation), plays it through automated test stations in real vehicles, and then evaluates the assistant’s answers with AI-based scoring — producing continuous, comparable quality statements across vehicle software versions. This position owns the evaluation and quality-measurement core of that platform: the part that decides whether an answer was good, and proves it consistently enough to gate a release.

Responsibilities

• Build and own the AI evaluation engine: discrete (e.g. 1–6) and continuous (0–100%) scoring with confidence values and partial scores, efficient evaluation via embeddings and cosine similarity, with selective LLM-as-judge only where it earns its cost and latency.
• Define and maintain the quality metric set for an automotive assistant: hallucination rate, answer faithfulness and relevance, intent-fulfilment, latency P95, BLEU / WER / CER — and defend their thresholds as release criteria.
• Develop generative test case and dialogue generation: LLM-based test case creation per domain (navigation, media, knowledge, car control) with configurable volume, complexity, scenario types and edge‑case coverage; automatic generation of test descriptions, expected dialogue flows, success criteria and evaluation prompts.
• Make generated dialogue realistically human: hesitations, filler words, incomplete sentences and colloquial speech instead of rigid scripts; generative multi‑turn conversations where follow‑up turns are produced dynamically from the assistant’s actual answer.
• Engineer test variability and reliability: randomised test parameters (language, voice characteristics, speaking rate, audio characteristics, dialects), complexity variants (short / nominal / complex) that preserve semantic intent, and automatic retry with clean separation of infrastructure failures from functional defects.
• Build prompt and model regression suites in LangFuse so prompt changes, model version upgrades and backend releases are caught before they reach vehicles — including version‑over‑version comparison and trend analysis.
• Contribute to audio simulation and multi‑speaker testing: realistic background noise (engine, road, music, phone calls, sirens, horns, passenger conversation), mixed‑language input, multi‑speaker simulation from different seat positions with position‑specific evaluation prompts, and comparison of synthetic against real native‑speaker recordings.
• Cover conventional test engineering alongside the AI layer: unit, integration and system tests across domain, adapter and infrastructure layers (pytest, testcontainers), automated smoke and release tests, and the supplier speech‑stack test suites.
• Support test execution in real vehicles and on the test rack (TISAX Level 3 environment), including device health checks, failover behaviour and triage of infrastructure versus assistant defects.
• Deliver reporting and analytics that non-engineers can act on: test statistics by batch and device, exports (PDF/CSV/Excel), version comparisons, performance trends and regression detection between software releases.
• Maintain traceability between test cases, requirements and defects, including versioned test cases with change history and diff views, and integration with the client’s test specification and requirements management systems.
• Coordinate with the backend, frontend, AI and hardware teams and with the client’s departments; report quality status to management in a form that supports go/no‑go decisions.

Skills

Must have

• 5+ years in test automation / quality engineering with strong Python, including ownership of a test framework rather than only writing cases in someone else’s.
• Hands‑on experience evaluating LLM or other non‑deterministic systems: building evaluation pipelines, LLM‑as‑judge patterns, embedding/cosine‑similarity scoring, and reasoning about scoring reliability and inter‑rater agreement.
• Practical command of LLM quality metrics — faithfulness, hallucination rate, relevance, plus at least one of BLEU / WER / CER — and the judgement to know what each does and does not prove.
• Experience with LangChain / LangGraph or equivalent, and with an LLM observability tool (LangFuse, Phoenix, Langsmith or comparable) for tracing, prompt versioning and evaluation runs.
• Prompt regression testing in practice: detecting quality drift across prompt edits and model version changes.
• Solid conventional test engineering: pytest, code coverage, test case design across positive, negative, boundary and regression cases; testcontainers or equivalent integration test tooling.
• Experience integrating ASR and TTS services (Azure Speech, Whisper, Cerence, Google TTS or equivalent), including streaming/real‑time behaviour.
• CI/CD integration of automated test suites (Azure DevOps Pipelines, GitHub Actions) and comfort with Docker; working knowledge of REST APIs and SQL/PostgreSQL for test data and results.
• English C1, and willingness to work onsite in the Ingolstadt/Munich area, including hands‑on test sessions in vehicles and on the test rack.

Nice to have Skills

• German B2 or above — strongly preferred; project language is German and test specifications and defect discussions often are too.
• Automotive voice assistant and vehicle interface experience; familiarity with in‑vehicle test benches, test racks or HiL environments.
• Audio engineering: background noise modelling, multi‑speaker and mixed‑language audio, speaking‑rate and dialect variation, synthetic versus human speech comparison.
• Test device and network infrastructure: remote device fleets, registration and health monitoring, failover mechanisms, reservation/scheduling systems.
• Multi‑language and localisation testing, including configurable wake‑words and language‑specific evaluation criteria.
• Awareness of TISAX, A‑SPICE, ISO/SAE 21434 and of EU AI Act documentation expectations for AI system testing.
• Experience evaluating safety‑relevant AI output where a wrong answer is more than a quality defect.