AI Test Engineer: Automotive LLM Evaluation & QA

Luxoft Germany; United States; Remote; USD 106,000 - 151,000; Full time

Job summary

DXC Luxoft in the Ingolstadt/Munich area seeks a seasoned QA Engineer to own AI evaluation and the test platform quality for in-vehicle voice assistants. You will design metrics, supervise end-to-end testing, and drive automation across embeddings, LLM scoring and audio pipelines.

You will coordinate with backend, frontend, AI and hardware teams, implement LangFuse-based regression suites, and deliver actionable reports on test statistics, trends and release readiness.

Qualifications

• 5+ years in test automation / quality engineering with strong Python, including ownership of a test framework.
• Hands-on experience evaluating LLM or other non-deterministic systems: building evaluation pipelines and scoring.
• Experience with LangChain / LangGraph and observability tools for tracing and versioning.
• Prompt regression testing: detecting quality drift across prompt edits and model versions.
• Solid CI/CD tooling experience and Docker; REST APIs and SQL/PostgreSQL for test data.

Responsibilities

• Build and own the AI evaluation engine with discrete and continuous scoring, embeddings, and confidence values.
• Define and maintain quality metrics for an automotive assistant: hallucination, faithfulness, relevance, latency, BLEU/WER/CER.
• Develop test case/dialogue generation across domains with dynamic, human-like dialogue.
• Engineer test variability and reliability; integrate audio/noise simulations and multi-speaker testing.
• Implement LangFuse-based regression suites to catch regressions across versions and releases.
• Coordinate with backend/frontend/AI/hardware teams and provide management-ready status reports.

Skills

Python

LLM evaluation

LangChain

LangGraph

pytest

Docker

REST APIs

PostgreSQL

Azure DevOps Pipelines

GitHub Actions

English (C1)

Tools

LangFuse

Langsmith

Phoenix

Job description

DXC Luxoft in the Ingolstadt/Munich area seeks a seasoned QA Engineer to own AI evaluation and the test platform quality for in-vehicle voice assistants. You will design metrics, supervise end-to-end testing, and drive automation across embeddings, LLM scoring and audio pipelines.

You will coordinate with backend, frontend, AI and hardware teams, implement LangFuse-based regression suites, and deliver actionable reports on test statistics, trends and release readiness.