Lead Consultant - QA(AI) Engineer
Logoslabs
Obtenez une réponse de cet employeur
Votre CV et votre lettre de motivation, adaptés exactement à ce que l’employeur recherche.
Avantages offerts par ce poste
Résumé du poste
Delphi is seeking a hands-on Senior QA Consultant to lead end-to-end testing of AI and GenAI applications across enterprise environments. You will own QA strategy, mentor a small QA pod, and collaborate with AI/ML, product, and business teams to deliver production-grade quality in regulated industries like finance, healthcare, and insurance.
The role requires strong automation and AI evaluation capabilities, experience with RAG pipelines, multi-agent systems, OCR, and speech-to-text solutions,
Qualifications
- 10–15 years of experience in Software QA / Test Engineering.
Responsabilités
- Own end-to-end quality assurance for AI/GenAI engagements across enterprise environments.
- Lead and mentor QA pods of 3–6 engineers across automation frameworks and delivery execution.
- Design and execute functional, API, backend, integration, regression, performance, and AI evaluation test cases.
- Build and maintain scalable automation frameworks for web, mobile, and API testing.
- Drive continuous quality improvements through reusable accelerators, frameworks, datasets, and evaluation tools.
- Define AI-specific test coverage including prompt validation, hallucination detection, groundedness checks, retrieval quality, tool-call correctness, latency, bias, and safety testing.
- Design and execute testing strategies for RAG architectures, multi-agent workflows, OCR systems, and speech-to-text applications.
- Build synthetic datasets covering edge cases, adversarial scenarios, persona-based workflows, and multimodal inputs.
- Implement LLM-as-a-Judge evaluation pipelines using customizable scoring rubrics and multi-model evaluations.
- Produce detailed AI quality scorecards and benchmarking reports across accuracy, relevancy, faithfulness, and safety metrics.
- Integrate AI evaluation and automation testing into Jenkins and Azure DevOps CI/CD pipelines.
- Build regression dashboards, quality gates, and automated reporting frameworks.
- Perform API testing, backend validation, database testing, and event-driven workflow validation.
- Conduct load and performance testing using modern performance engineering tools.
- Support cross-browser, parallel, and scalable automation execution environments.
- Act as the client-facing QA lead for enterprise engagements.
- Conduct quality reviews, risk assessments, and stakeholder reporting sessions.
- Translate technical quality metrics into business-focused outcomes and recommendations.
- Collaborate with engineering, AI/ML, product, and leadership teams to ensure successful delivery outcomes.
- Strong leadership and mentoring capabilities.
- Excellent verbal and written communication skills.
- Client-centric and solution-oriented mindset.
- Strong stakeholder management and collaboration abilities.
- Ownership mindset with proactive problem-solving approach.
- Ability to work in fast-paced consulting environments.
- Continuous learner with strong interest in emerging AI testing methodologies and tools.
- Bachelor’s degree in Computer Science, Engineering, or a related field
- 7
Connaissances
Formation
Outils
Description du poste
At Delphi, we believe in creating an environment where our people thrive. Our hybrid work model empowers you to choose where you work - whether it's from the office, your home, or a mix of both - so you can prioritize what matters most. We are committed to supporting your personal goals, family, and overall well-being while driving transformative results for our clients.
We welcome exceptional talent from anywhere across the globe. Interviews and onboarding are conducted virtually, reflecting our digital-first mindset.
Rooted in the region, we specialize in delivering tailored, impactful solutions in Data, Advanced Analytics and AI, Infrastructure, Cloud Security, and Application Modernization. Whether it’s enabling predictive analytics, transforming operations with automation, or driving customer engagement with intelligent platforms, we are the trusted partner for organizations ready to embrace a smarter, more efficient future.
We are looking for a hands‑on Senior QA Consultant to lead end‑to‑end testing of AI and generative AI applications across enterprise environments. This role combines strong expertise in traditional QA engineering with modern AI evaluation and validation practices. The ideal candidate will drive quality assurance initiatives for RAG pipelines, multi‑agent systems, OCR and speech‑to‑text solutions while ensuring production‑grade quality outcomes in regulated industries such as finance, healthcare, and insurance. The successful candidate will lead a small QA pod, collaborate closely with engineering, AI/ML, product, and business teams, and act as the client‑facing QA owner for enterprise AI engagements. This role requires strong technical leadership, automation expertise, AI evaluation capabilities, and excellent stakeholder management skills.
- 10–15 years of experience in Software QA / Test Engineering
- 3–4 years of experience leading QA teams or delivery pods
- Hands‑on experience testing AI / GenAI applications in production environments
- Experience working within regulated domains such as finance, healthcare, insurance, or similar industries
- Own end‑to‑end quality assurance for AI/GenAI engagements including test strategy, planning, execution, defect management, reporting, and exit criteria
- Lead and mentor QA pods of 3–6 engineers across automation frameworks, AI testing practices, and delivery execution
- Design and execute functional, API, backend, integration, regression, performance, and AI evaluation test cases
- Build and maintain scalable automation frameworks for web, mobile, and API testing
- Drive continuous quality improvements through reusable accelerators, frameworks, datasets, and evaluation tools
- Define AI‑specific test coverage including prompt validation, hallucination detection, groundedness checks, retrieval quality, tool‑call correctness, latency, bias, and safety testing
- Design and execute testing strategies for RAG architectures, multi‑agent workflows, OCR systems, and speech‑to‑text applications
- Build synthetic datasets covering edge cases, adversarial scenarios, persona‑based workflows, and multimodal inputs
- Implement LLM‑as‑a‑Judge evaluation pipelines using customizable scoring rubrics and multi‑model evaluations
- Produce detailed AI quality scorecards and benchmarking reports across accuracy, relevancy, faithfulness, and safety metrics
- Integrate AI evaluation and automation testing into Jenkins and Azure DevOps CI/CD pipelines
- Build regression dashboards, quality gates, and automated reporting frameworks
- Perform API testing, backend validation, database testing, and event‑driven workflow validation
- Conduct load and performance testing using modern performance engineering tools
- Support cross‑browser, parallel, and scalable automation execution environments
- Act as the client‑facing QA lead for enterprise engagements
- Conduct quality reviews, risk assessments, and stakeholder reporting sessions
- Translate technical quality metrics into business‑focused outcomes and recommendations
- Collaborate with engineering, AI/ML, product, and leadership teams to ensure successful delivery outcomes
- Functional Testing, Exploratory Testing, Requirements Traceability, and Test Design Techniques
- API Testing using REST, GraphQL, Postman/Newman, RestAssured, and Requests
- Backend and Database Testing across SQL Server, MySQL, Oracle, PostgreSQL, and ETL/Data Pipelines
- Automation Frameworks using Selenium, Playwright, PyTest, and Appium
- Performance Testing using JMeter, k6, or Locust
- Strong programming expertise in Python or Java
- Knowledge of OOP concepts, design patterns, data structures, and unit testing frameworks
- Ability to build utilities, custom assertions, CLI tools, and reusable automation components
- Hands‑on experience testing AI/GenAI applications in enterprise environments
- Knowledge of LLM platforms including OpenAI, Anthropic, Azure OpenAI, AWS Bedrock, and Vertex AI
- Experience with orchestration frameworks such as LangChain, LlamaIndex, and Semantic Kernel
- Understanding of multi‑agent systems, memory handling, routing, tool calling, and guardrails
- Expertise in RAG evaluation, vector databases, embeddings, retrieval quality, and hallucination testing
- Familiarity with AI evaluation platforms such as RAGAS, DeepEval, Galileo, Arize Phoenix, Patronus, Okareo, or similar tools
- Experience validating OCR systems and speech‑to‑text solutions using WER/CER and degradation testing methodologies
- Azure DevOps, Jenkins, JIRA/Xray or Zephyr, GitHub/GitLab/Bitbucket
- CI/CD pipelines, regression dashboards, and quality gate implementation
- Exposure to Power Platform tools including Power Apps, Power BI, and Power Automate is a plus
- Experience working in regulated industry environments such as banking, financial services, healthcare, insurance, or life sciences
- Understanding of compliance requirements including data privacy, PII/PHI handling, explainability, and AI governance practices
- Strong leadership and mentoring capabilities
- Excellent verbal and written communication skills
- Client‑centric and solution‑oriented mindset
- Strong stakeholder management and collaboration abilities
- Ownership mindset with proactive problem‑solving approach
- Ability to work effectively in fast‑paced consulting environments
- Continuous learner with strong interest in emerging AI testing methodologies and tools
- Bachelor’s degree in Computer Science, Engineering, or a related field
- Master’s degree preferred
- Certifications such as ISTQB Advanced, Certified Agile Tester, AWS/Azure AI Certifications, or equivalent
- Experience with adversarial testing and responsible AI frameworks such as NIST AI RMF or ISO 42001
- Exposure to MLOps / LLMOps observability and production monitoring
- Contributions to QA or AI communities through blogs, speaking sessions, or open source initiatives
- High‑quality delivery outcomes with measurable reduction in defect leakage
- Successful implementation of reusable AI QA accelerators and automation frameworks
- Strong stakeholder and client feedback across engagements
- Growth and technical enablement of QA pod members under leadership
- Continuous improvement of AI testing practices, evaluation frameworks, and delivery standards
At Delphi, we are dedicated to creating an environment where you can thrive, both professionally and personally. Our competitive compensation package, performance‑based incentives, and health benefits are designed to ensure you’re well‑supported. We believe in your continuous growth and offer company‑sponsored certifications, training programs, and skill‑building opportunities to help you succeed.
We foster a culture of inclusivity and support, with remote work options and a fully supported work‑from‑home setup to ensure your comfort and productivity. Our positive and inclusive culture includes team activities, wellness and mental health programs to ensure you feel supported.
'/%3e%3cpath%20d='M101.744%2012.4726L102.894%2016.1947L99.0021%2013.2201L101.744%2012.4726ZM105.3%208.40937H100.488L99.0026%203.59473L97.5116%208.41047L92.7%208.40385L96.5965%2011.3834L95.1055%2016.1942L99.0021%2013.2196L101.408%2011.3834L105.3%208.40992V8.40937Z'%20fill='white'/%3e%3cdefs%3e%3clinearGradient%20id='paint0_linear_5593_1381'%20x1='90'%20y1='9.44471'%20x2='99'%20y2='9.44471'%20gradientUnits='userSpaceOnUse'%3e%3cstop%20stop-color='%2300AD70'/%3e%3cstop%20offset='0.998594'%20stop-color='%2300B67A'/%3e%3cstop%20offset='1'%20stop-color='%23DCDCE6'/%3e%3c/linearGradient%3e%3c/defs%3e%3c/svg%3e){kind=small}
