AI Solutions QA Analyst - Remote-First Legal AI Quality

McDermott Will & Schulte

Wilmington (DE)

Remote

USD 93,000 - 142,000

Full time

3 days ago

Be an early applicant

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

McDermott Will & Schulte is building a leading AI QA program in the legal tech space. The AI Solutions QA Analyst will own the full quality lifecycle for AI-powered solutions, from unit tests to system and scalability testing, with a strong emphasis on AI output validation.

You will partner with AI Solutions Analysts and engineering to design test cases using AI tools, implement automated suites in Azure DevOps CI/CD, and translate security guidance into concrete test plans while maintaining

Qualifications

AI fluency — you use AI tools actively and can direct leading AI platforms to generate, evaluate, and refine test cases
4+ years of formal QA experience, including writing and executing test scripts across the full testing lifecycle
Strong experience with unit, integration, system, and regression testing — both manual and automated
Hands-on experience integrating test suites into CI/CD pipelines; Azure DevOps experience strongly preferred
Demonstrated ability to design boundary and edge case tests for complex, data-driven applications
Understanding of performance and scalability testing concepts; experience with load testing tools a plus
Solid grasp of defect tracking, test documentation, and quality metrics reporting
Strong analytical thinking — you approach untested AI behavior the way an engineer approaches an unknown system
BS in Computer Science, Information Systems, Engineering, or equivalent experience

Responsibilities

Own the full QA lifecycle for AI-powered solutions — from unit and integration testing through system, performance, and scalability testing
Partner with Analysts to generate and evaluate test cases using AI-assisted tools, with particular focus on boundary conditions, edge cases, and AI output validation
Work with the engineering team to define and integrate automated test suites into the Azure DevOps CI/CD pipeline for each solution
Define system-level test strategies in collaboration with the Director of QA, covering functional, regression, and non-functional testing requirements
Coordinate with the dedicated security team to translate security testing guidance into solution-specific security test cases
Evaluate AI solution reliability across varied inputs — ensuring consistent, predictable behavior under real-world conditions
Maintain test documentation, defect tracking, and quality metrics that give leadership clear visibility into solution readiness
Champion a quality culture across the delivery team — making testing a natural part of the development workflow, not a gate at the end