Senior Test Engineer – AI Voice Systems

Project description

Our client is advancing its in-vehicle voice assistant into an intelligent, AI-powered companion. Large-language-model capabilities (Azure OpenAI / ChatGPT) have been running in production across vehicles with new E³-architecture models featuring enhanced voice functions from the factory. The customer backend is the cloud AI orchestration service behind this: it receives requests from the vehicle, classifies and routes them, orchestrates the LLM, tool services and specialised agents, and returns an answer or action to the car. DXC Luxoft serves as the end-to-end delivery partner, working in a joint product team with the client's engineers on the Azure platform (AKS, Azure OpenAI, AI Foundry, Managed Identity, Azure Monitor, Azure DevOps). This role sits in the end-to-end testing and AI evaluation work package — the quality gate for the whole platform. LLM-based assistants generate variable, context-dependent answers: every response is unique and only conditionally predictable, so classical test systems with rigid expected values cannot cover them. The work package therefore builds and operates an intelligent, AI-supported E2E test platform that reacts dynamically to each assistant output and continues the test dialogue appropriately. Concretely, the platform generates synthetic speech with variable parameters (languages, pitch, emotional colouring, natural variation), plays it through automated test stations in real vehicles, and then evaluates the assistant's answers with AI-based scoring — producing continuous, comparable quality statements across vehicle software versions. This position owns the evaluation and quality-measurement core of that platform: the part that decides whether an answer was good, and proves it consistently enough to gate a release.

Responsibilities

• Build and own the AI evaluation engine: discrete (e.g. 1–6) and continuous (0–100%) scoring with confidence values and partial scores, efficient evaluation via embeddings and cosine similarity, with selective LLM-as-judge only where it earns its cost and latency.
• Define and maintain the quality metric set for an automotive assistant: hallucination rate, answer faithfulness and relevance, intent-fulfilment, latency P95, BLEU / WER / CER — and defend their thresholds as release criteria.
• Develop generative test case and dialogue generation: LLM-based test case creation per domain (navigation, media, knowledge, car control) with configurable volume, complexity, scenario types and edge-case coverage; automatic generation of test descriptions, expected dialogue flows, success criteria and evaluation prompts.
• Make generated dialogue realistically human: hesitations, filler words, incomplete sentences and colloquial speech instead of rigid scripts; generative multi-turn conversations where follow-up turns are produced dynamically from the assistant's actual answer.
• Engineer test variability and reliability: randomised test parameters (language, voice characteristics, speaking rate, audio characteristics, dialects), complexity variants (short / nominal / complex) that preserve semantic intent, and automatic retry with clean separation of infrastructure failures from functional defects.
• Build prompt and model regression suites in LangFuse so prompt changes, model version upgrades and backend releases are caught before they reach vehicles — including version-over-version comparison and trend analysis.
• Contribute to audio simulation and multi-speaker testing: realistic background noise (engine, road, music, phone calls, sirens, horns, passenger conversation), mixed-language input, multi-speaker simulation from different seat positions with position-specific evaluation prompts, and comparison of synthetic against real native-speaker recordings.
• Cover conventional test engineering alongside the AI layer: unit, integration and system tests across domain, adapter and infrastructure layers (pytest, testcontainers), automated smoke and release tests, and the supplier speech-stack test suites.
• Support test execution in real vehicles and on the test rack (TISAX Level 3 environment), including device health checks, failover behaviour and triage of infrastructure versus assistant defects.
• Deliver reporting and analytics that non-engineers can act on: test statistics by batch and device, exports (PDF/CSV/Excel), version comparisons, performance trends and regression detection between software releases.
• Maintain traceability between test cases, requirements and defects, including versioned test cases with change history and diff views, and integration with the client's test specification and requirements management systems.
• Coordinate with the backend, frontend, AI and hardware teams and with the client's departments; report quality status to management in a form that supports go/no-go decisions.

SKILLS

Must have

• 5+ years in test automation / quality engineering with strong Python, including ownership of a test framework rather than only writing cases in someone else's.
• Hands-on experience evaluating LLM or other non-deterministic systems: building evaluation pipelines, LLM-as-judge patterns, embedding/cosine-similarity scoring, and reasoning about scoring reliability and inter-rater agreement.
• Practical command of LLM quality metrics — faithfulness, hallucination rate, relevance, plus at least one of BLEU / WER / CER — and the judgement to know what each does and does not prove.
• Experience with LangChain / LangGraph or equivalent, and with an LLM observability tool (LangFuse, Phoenix, Langsmith or comparable) for tracing, prompt versioning and evaluation runs.
• Prompt regression testing in practice: detecting quality drift across prompt edits and model version changes.
• Solid conventional test engineering: pytest, code coverage, test case design across positive, negative, boundary and regression cases; testcontainers or equivalent integration test tooling.
• Experience integrating ASR and TTS services (Azure Speech, Whisper, Cerence, Google TTS or equivalent), including streaming/real-time behaviour.
• CI/CD integration of automated test suites (Azure DevOps Pipelines, GitHub Actions) and comfort with Docker; working knowledge of REST APIs and SQL/PostgreSQL for test data and results.
• English C1, and willingness to work onsite in the Ingolstadt/Munich area, including hands-on test sessions in vehicles and on the test rack.

Nice to have

• German B2 or above — strongly preferred; project language is German and test specifications and defect discussions often are too. • Automotive voice assistant and vehicle interface experience; familiarity with in-vehicle test benches, test racks or HiL environments. • Audio engineering: background noise modelling, multi-speaker and mixed-language audio, speaking-rate and dialect variation, synthetic versus human speech comparison. • Frontend skills for reporting and review tooling (Vue / React / Angular): dashboards, data visualisation, PDF/CSV/Excel export, browser-based remote test control. • Test device and network infrastructure: remote device fleets, registration and health monitoring, failover mechanisms, reservation/scheduling systems. • Multi-language and localisation testing, including configurable wake-words and language-specific evaluation criteria. • Awareness of TISAX, A-SPICE, ISO/SAE 21434 and of EU AI Act documentation expectations for AI system testing. • Experience evaluating safety-relevant AI output where a wrong answer is more than a quality defect.