Senior AI QA Lead for LLM Safety & Pedagogy

SoftServe; Chile; Presencial; CLP 18.000.000 - 30.000.000; Jornada completa

Descripción de la vacante

SoftServe is seeking a Senior QC Engineer to own quality across an AI Student Tutor project, coordinating with AI/ML, product, and data teams. You will drive test strategy, acceptance criteria, and UAT with stakeholders in the US and Europe, ensuring safety, pedagogy, and compliance across scenarios.

You will design and execute functional, integration, and performance tests, validate prompt governance, and reproduce issues for fast remediation, while maintaining clear documentation and strong

Formación

• Proven experience leading QA/QC for complex, multi-workstream software programs, including authoring test strategies and acceptance test plans from scratch.
• Experience testing AI/LLM-based or conversational agent systems — prompt-driven behavior, RAG/grounding validation, model-output quality rubrics, and safety/guardrail testing.
• Strong written documentation skills for test plans, traceability matrices, defect reports, and readiness packages.
• Strong client-facing communication skills, comfortable running bug-verification calls/sessions directly with client stakeholders and writing defect descriptions clear enough for a remote team to act on without follow-up questions.
• Availability during the US-Europe daily overlap window to support live client bug verification and same-day handoff to the Europe-based team.

Responsabilidades

• Own creation of the Acceptance Testing Plan (ATP), defining acceptance criteria, test types, roles and responsibilities, test data, evidence expectations, defect severity definitions, retest process, and formal sign-off requirements, in coordination with Covista.
• Design and execute the overall test strategy spanning functional, integration, regression, load/performance, accessibility, safety, and pedagogical-quality testing.
• Test the AI agent application itself, including single-assistant routing/orchestration logic, prompt governance and version-tracking, persona-adaptive behavior, content-validation and grounding/citation pipelines, and resistance to identity-override and prompt-injection attempts.
• Run safety scenario testing across low, medium, high, and critical distress-handling cases, measuring detection accuracy and false-negative/false-positive rates.
• Drive UAT: build test scripts, execute with client's Product, SMEs, IT, Security/Compliance, and pilot stakeholders; manage defect triage, disposition, and re-test.
• Serve as the primary client-facing point of contact for quality matters: communicate testing status, defect trends, and pilot-readiness risk directly to client's Product, SMEs, and other UAT stakeholders.
• Jointly verify and reproduce reported bugs with Covista during UAT and Hypercare, confirming severity, scope, and expected vs. actual behavior before logging.
• Translate verified client-reported issues into clear, actionable defect write-ups (repro steps, environment, evidence, severity) and hand them off to the Europe-based delivery team, working within the daily US-Europe overlap window to ensure timely triage and unblock next-day resolution.

Conocimientos

QA leadership

AI/LLM testing

Documentation

Client-facing communication

US-Europe overlap availability

LMS integration testing awareness

Descripción del empleo

SoftServe is seeking a Senior QC Engineer to own quality across an AI Student Tutor project, coordinating with AI/ML, product, and data teams. You will drive test strategy, acceptance criteria, and UAT with stakeholders in the US and Europe, ensuring safety, pedagogy, and compliance across scenarios.

You will design and execute functional, integration, and performance tests, validate prompt governance, and reproduce issues for fast remediation, while maintaining clear documentation and strong