Sr. QA Specialist

Florida Blue; United States; On-site; USD 120,000 - 160,000; Full time

Job summary

Florida Blue, in partnership with Magnit, seeks a Senior QA Engineer to lead testing of conversational AI applications in a healthcare context. You will design test strategies, validate prompts, and build automation for APIs and UI.

You will collaborate with AI/ML teams to address non-deterministic outputs, hallucinations and bias, and ensure HIPAA-aligned security testing. This contract role offers exposure to GuideWell affiliates.

Qualifications

• 5+ years of QA experience, with AI/ML testing experience preferred.
• Experience designing, implementing, and maintaining automated test suites.
• Knowledge of API testing and CI/CD pipelines.
• Proficiency in at least one programming language (Python/Java/JavaScript).
• Understanding of AI concepts, LLMs, NLP and failure modes.

Responsibilities

• Lead QA efforts for conversational AI applications and ensure reliability, accuracy, and security.
• Design and implement comprehensive test strategies and edge-case scenarios for AI behavior.
• Develop and maintain automated test suites for AI workflows, including API and UI testing.
• Validate prompts and prompt injection/safety considerations with AI/ML teams.
• Document defects and quality metrics; communicate results to stakeholders.

Skills

QA testing

AI/ML testing

Test automation

API testing

CI/CD

Prompt engineering

Python/Java/JS

Education

Bachelor's Degree in IT or related field

Tools

Selenium

Cypress

Postman

REST Assured

Job description

This is a contract opportunity. You will be employed by Magnit to work on assignment with GuideWell and subsidiary companies including Florida Blue.

We are seeking an experienced Senior QA Engineer to lead quality assurance efforts for our conversational AI applications. This role requires deep expertise in testing AI-driven systems, prompt engineering validation, and test automation. You will be responsible for designing comprehensive test strategies, identifying edge cases in AI behavior, and ensuring our conversational AI products meet the highest standards of reliability, accuracy, and security.

Key Responsibilities

• AI Application Testing: Design and execute test plans specifically for conversational AI systems, including chatbots, virtual assistants, and language model-based applications
• Prompt Engineering Validation: Collaborate with AI/ML teams to test and validate prompt effectiveness, consistency, and safety across various user scenarios
• Test Automation Development: Build and maintain automated test suites for conversational AI workflows, including API testing, response validation, and regression testing
• Quality Strategy: Develop QA strategies that address unique challenges of AI testing, including non-deterministic outputs, hallucination detection, and bias identification
• Security & Compliance: Ensure testing aligns with organizational security standards, including HIPAA compliance (Standard 4500), information protection requirements (Standard 5110), and access control protocols
• Documentation & Reporting: Create detailed test cases, defect reports, and quality metrics; communicate findings to stakeholders
• Cross-functional Collaboration: Work with product, engineering, and AI/ML teams to understand requirements and validate solutions

Required Qualifications

• 5+ years of QA experience, with at least 2+ years testing AI/ML-based applications or conversational systems
• Prompt Engineering Expertise: Demonstrated experience designing, testing, and validating prompts for language models; understanding of prompt injection, jailbreaking, and safety considerations
• Test Automation Proficiency: Strong hands‑on experience with test automation frameworks and tools (e.g., Selenium, Cypress, Postman, REST Assured, or equivalent); ability to write clean, maintainable automation code
• Technical Skills: Proficiency in at least one programming language (Python, Java, JavaScript, or similar); familiarity with API testing and CI/CD pipelines
• AI/Conversational AI Knowledge: Understanding of how conversational AI systems work, including LLMs, NLP concepts, and common failure modes
• Analytical Mindset: Strong problem‑solving skills with ability to identify edge cases and non‑obvious failure scenarios in AI systems
• Communication: Excellent written and verbal communication skills; ability to explain technical QA concepts to non‑technical stakeholders

Preferred Qualifications

• Experience testing multiple AI platforms (OpenAI, Google Vertex AI, Azure OpenAI, Anthropic Claude, etc.)
• Familiarity with AI safety testing frameworks and bias detection methodologies
• Experience with performance and load testing for AI applications
• Knowledge of HIPAA, SOC 2, or other healthcare/regulated industry compliance requirements
• Certification in QA, test automation, or related field
• Experience with agile/scrum methodologies
• Background in healthcare, financial services, or other regulated industries

Technical Competencies

• Test case design and execution
• Test automation and scripting
• API and integration testing
• Performance and load testing
• Defect tracking and reporting
• SQL and database query basics
• Version control systems (Git)
• Prompt engineering and LLM interaction
• Security and compliance testing

Required Experience

• 5-8 years of Professional experience in an IT related field preferably in a development or testing environment

Required Education

• Bachelor’s Degree in an IT related field or relevant work experience

GuideWell and its family of companies has partnered with Magnit as its Managed Service Provider (MSP) and Employer of Record (EOR) since 2018. In May of 2025, GuideWell joined Magnit Direct Source to implement the GuideWell Contractor Cohort. This program is designed to create, manage and curate a contractor talent pool for temporary contract opportunities with GuideWell. As a contractor working on temporary assignment with GuideWell, you'll be employed by Magnit. You'll have the opportunity to work on temporary projects at GuideWell companies that make a real difference in people's lives while enjoying the benefits of being part of the Magnit team.