Gen AI QA Engineer: LLM & AI Testing Expert

Synechron; Charlotte (NC); On-site; USD 90,000 - 100,000; Full time

Benefits offered by this job

Paid annual leave

Paternity leave

Maternity leave

Insurance plan

Retirement savings

Udemy for Business

Commuter benefits

Training opportunities

Global offices

Job summary

Synechron is seeking a Gen AI Tester to ensure the quality, reliability, security, and performance of enterprise Generative AI applications and LLM solutions. The role requires 6+ years in QA and 2+ years testing AI-powered systems, with hands-on skills in Python, Playwright/Selenium, and API testing.

You will validate AI responses, assess risk, and help integrate AI QA into CI/CD pipelines. The position focuses on AI governance, security testing, and collaboration across AI Engineers, Data

Qualifications

• 6+ years of experience in Quality Assurance, Test Automation, or Quality Engineering.
• 2+ years of hands-on experience testing Generative AI, LLM-based applications, or AI-powered enterprise solutions.
• Strong understanding of Generative AI, Large Language Models (LLMs), Prompt Engineering, and AI application architectures.
• Experience with AI platforms such as Azure OpenAI, OpenAI GPT, Anthropic Claude, Google Gemini, or Meta Llama.
• Hands-on experience testing AI chatbots, virtual assistants, AI agents, and RAG-based applications.
• Strong expertise in automation testing using Playwright, Selenium, Cypress, or similar frameworks.

Responsibilities

• Design and execute comprehensive test strategies for Generative AI, LLM-powered applications, AI agents, and RAG-based solutions.
• Validate AI-generated responses for accuracy, relevance, consistency, factual correctness, and business alignment.
• Develop and maintain test cases for prompt engineering, prompt optimization, output validation, and conversational AI workflows.
• Perform functional, integration, regression, end-to-end, and user acceptance testing of AI-powered applications.
• Evaluate model performance for hallucinations, bias, toxicity, fairness, grounding accuracy, and response quality.
• Test Retrieval-Augmented Generation (RAG) systems, vector search implementations, and enterprise knowledge repositories.
• Build and maintain automated testing frameworks using Python, Playwright, Selenium, PyTest, or Robot Framework.
• Conduct API testing for AI services, LLM endpoints, orchestration layers, and enterprise integrations.
• Perform AI security testing, including prompt injection, jailbreak testing, adversarial prompts, and data leakage validation.
• Validate AI governance controls, responsible AI guardrails, and compliance requirements.
• Collaborate with AI Engineers, Data Scientists, Product Owners, and Business Stakeholders to improve model quality and application reliability.
• Integrate AI quality assurance processes into CI/CD pipelines and deployment workflows.
• Document defects, testing results, risk assessments, and recommendations for continuous improvement.
• Stay current with emerging AI testing methodologies, tools, and industry best practices.

Skills

Quality Assurance

Test Automation

Generative AI testing

LLM evaluation

Python automation

API testing

Azure AI

SQL testing

CI/CD

Stakeholder mgmt

Tools

Playwright

Selenium

Cypress

Postman

Swagger/OpenAPI

LangSmith

TruLens

Job description

Synechron is seeking a Gen AI Tester to ensure the quality, reliability, security, and performance of enterprise Generative AI applications and LLM solutions. The role requires 6+ years in QA and 2+ years testing AI-powered systems, with hands-on skills in Python, Playwright/Selenium, and API testing.

You will validate AI responses, assess risk, and help integrate AI QA into CI/CD pipelines. The position focuses on AI governance, security testing, and collaboration across AI Engineers, Data