Senior AI QA Lead — GenAI for Enterprise (Remote)

Logoslabs; San Francisco (CA); Hybrid; USD 120,000 - 180,000; Full time

Benefits offered by this job

Remote work options

Company-sponsored certifications

Training programs

Health benefits

Job summary

Delphi is seeking a hands-on Senior QA Consultant to lead end-to-end testing of AI and GenAI applications across enterprise environments. You will own QA strategy, mentor a small QA pod, and collaborate with AI/ML, product, and business teams to deliver production-grade quality in regulated industries like finance, healthcare, and insurance.

The role requires strong automation and AI evaluation capabilities, experience with RAG pipelines, multi-agent systems, OCR, and speech-to-text solutions,

Qualifications

• 10–15 years of experience in Software QA / Test Engineering.

Responsibilities

• Own end-to-end quality assurance for AI/GenAI engagements across enterprise environments.
• Lead and mentor QA pods of 3–6 engineers across automation frameworks and delivery execution.
• Design and execute functional, API, backend, integration, regression, performance, and AI evaluation test cases.
• Build and maintain scalable automation frameworks for web, mobile, and API testing.
• Drive continuous quality improvements through reusable accelerators, frameworks, datasets, and evaluation tools.
• Define AI-specific test coverage including prompt validation, hallucination detection, groundedness checks, retrieval quality, tool-call correctness, latency, bias, and safety testing.
• Design and execute testing strategies for RAG architectures, multi-agent workflows, OCR systems, and speech-to-text applications.
• Build synthetic datasets covering edge cases, adversarial scenarios, persona-based workflows, and multimodal inputs.
• Implement LLM-as-a-Judge evaluation pipelines using customizable scoring rubrics and multi-model evaluations.
• Produce detailed AI quality scorecards and benchmarking reports across accuracy, relevancy, faithfulness, and safety metrics.
• Integrate AI evaluation and automation testing into Jenkins and Azure DevOps CI/CD pipelines.
• Build regression dashboards, quality gates, and automated reporting frameworks.
• Perform API testing, backend validation, database testing, and event-driven workflow validation.
• Conduct load and performance testing using modern performance engineering tools.
• Support cross-browser, parallel, and scalable automation execution environments.
• Act as the client-facing QA lead for enterprise engagements.
• Conduct quality reviews, risk assessments, and stakeholder reporting sessions.
• Translate technical quality metrics into business-focused outcomes and recommendations.
• Collaborate with engineering, AI/ML, product, and leadership teams to ensure successful delivery outcomes.
• Strong leadership and mentoring capabilities.
• Excellent verbal and written communication skills.
• Client-centric and solution-oriented mindset.
• Strong stakeholder management and collaboration abilities.
• Ownership mindset with proactive problem-solving approach.
• Ability to work in fast-paced consulting environments.
• Continuous learner with strong interest in emerging AI testing methodologies and tools.
• Bachelor’s degree in Computer Science, Engineering, or a related field
• 7

Skills

QA testing

AI/GenAI testing

Leadership

Regulated domains

Automation frameworks

Education

Bachelor’s degree in Computer Science or related field

Master’s degree preferred

Tools

Selenium

Playwright

PyTest

Appium

JMeter

k6

Locust

Jenkins

Azure DevOps

Postman

RestAssured

Job description

Delphi is seeking a hands-on Senior QA Consultant to lead end-to-end testing of AI and GenAI applications across enterprise environments. You will own QA strategy, mentor a small QA pod, and collaborate with AI/ML, product, and business teams to deliver production-grade quality in regulated industries like finance, healthcare, and insurance.

The role requires strong automation and AI evaluation capabilities, experience with RAG pipelines, multi-agent systems, OCR, and speech-to-text solutions,