Gen AI Tester

Synechron; Charlotte (NC); On-site; USD 90,000 - 100,000; Full time

Benefits offered by this job

Paid annual leave

Paternity leave

Maternity leave

Insurance plan

Retirement savings

Udemy for Business

Commuter benefits

Training opportunities

Global offices

Job summary

Synechron is seeking a Gen AI Tester to ensure the quality, reliability, security, and performance of enterprise Generative AI applications and LLM solutions. The role requires 6+ years in QA and 2+ years testing AI-powered systems, with hands-on skills in Python, Playwright/Selenium, and API testing.

You will validate AI responses, assess risk, and help integrate AI QA into CI/CD pipelines. The position focuses on AI governance, security testing, and collaboration across AI Engineers, Data

Qualifications

• 6+ years of experience in Quality Assurance, Test Automation, or Quality Engineering.
• 2+ years of hands-on experience testing Generative AI, LLM-based applications, or AI-powered enterprise solutions.
• Strong understanding of Generative AI, Large Language Models (LLMs), Prompt Engineering, and AI application architectures.
• Experience with AI platforms such as Azure OpenAI, OpenAI GPT, Anthropic Claude, Google Gemini, or Meta Llama.
• Hands-on experience testing AI chatbots, virtual assistants, AI agents, and RAG-based applications.
• Strong expertise in automation testing using Playwright, Selenium, Cypress, or similar frameworks.

Responsibilities

• Design and execute comprehensive test strategies for Generative AI, LLM-powered applications, AI agents, and RAG-based solutions.
• Validate AI-generated responses for accuracy, relevance, consistency, factual correctness, and business alignment.
• Develop and maintain test cases for prompt engineering, prompt optimization, output validation, and conversational AI workflows.
• Perform functional, integration, regression, end-to-end, and user acceptance testing of AI-powered applications.
• Evaluate model performance for hallucinations, bias, toxicity, fairness, grounding accuracy, and response quality.
• Test Retrieval-Augmented Generation (RAG) systems, vector search implementations, and enterprise knowledge repositories.
• Build and maintain automated testing frameworks using Python, Playwright, Selenium, PyTest, or Robot Framework.
• Conduct API testing for AI services, LLM endpoints, orchestration layers, and enterprise integrations.
• Perform AI security testing, including prompt injection, jailbreak testing, adversarial prompts, and data leakage validation.
• Validate AI governance controls, responsible AI guardrails, and compliance requirements.
• Collaborate with AI Engineers, Data Scientists, Product Owners, and Business Stakeholders to improve model quality and application reliability.
• Integrate AI quality assurance processes into CI/CD pipelines and deployment workflows.
• Document defects, testing results, risk assessments, and recommendations for continuous improvement.
• Stay current with emerging AI testing methodologies, tools, and industry best practices.

Skills

Quality Assurance

Test Automation

Generative AI testing

LLM evaluation

Python automation

API testing

Azure AI

SQL testing

CI/CD

Stakeholder mgmt

Tools

Playwright

Selenium

Cypress

Postman

Swagger/OpenAPI

LangSmith

TruLens

Job description

At Synechron, we believe in the power of digital to transform businesses for the better. Our global consulting firm combines creativity and innovative technology to deliver industry-leading digital solutions. Synechron’s progressive technologies and optimization strategies span end-to-end Artificial Intelligence, Consulting, Digital, Cloud & DevOps, Data, and Software Engineering, servicing an array of noteworthy financial services and technology firms. Through research and development initiatives in our FinLabs we develop solutions for modernization, from Artificial Intelligence and Blockchain to Data Science models, Digital Underwriting, mobile-first applications and more. Over the last 20+ years, our company has been honored with multiple employer awards, recognizing our commitment to our talented teams. With top clients to boast about, Synechron has a global workforce of 16,700+, and has 57 offices in 22 countries within key global markets.

We are seeking a highly skilled Gen AI Tester to ensure the quality, reliability, security, and performance of enterprise Generative AI applications and Large Language Model (LLM) solutions. The ideal candidate will possess strong experience in AI testing, prompt validation, LLM evaluation, automation testing, API testing, and AI risk assessment. This role offers an exciting opportunity to work on cutting‑edge AI initiatives, validating AI‑enabled applications used across Banking, Financial Services, and other enterprise domains.

Additional Information

The base salary for this position will vary based on geography and other factors. In accordance with law, the base salary for this role if filled within Charlotte, NC is $90K - $100K/year & benefits (see below).

The Role

Responsibilities

• Design and execute comprehensive test strategies for Generative AI, LLM‑powered applications, AI agents, and RAG‑based solutions.
• Validate AI‑generated responses for accuracy, relevance, consistency, factual correctness, and business alignment.
• Develop and maintain test cases for prompt engineering, prompt optimization, output validation, and conversational AI workflows.
• Perform functional, integration, regression, end‑to‑end, and user acceptance testing of AI‑powered applications.
• Evaluate model performance for hallucinations, bias, toxicity, fairness, grounding accuracy, and response quality.
• Test Retrieval‑Augmented Generation (RAG) systems, vector search implementations, and enterprise knowledge repositories.
• Build and maintain automated testing frameworks using Python, Playwright, Selenium, PyTest, or Robot Framework.
• Conduct API testing for AI services, LLM endpoints, orchestration layers, and enterprise integrations.
• Perform AI security testing, including prompt injection, jailbreak testing, adversarial prompts, and data leakage validation.
• Validate AI governance controls, responsible AI guardrails, and compliance requirements.
• Collaborate with AI Engineers, Data Scientists, Product Owners, and Business Stakeholders to improve model quality and application reliability.
• Integrate AI quality assurance processes into CI/CD pipelines and deployment workflows.
• Document defects, testing results, risk assessments, and recommendations for continuous improvement.
• Stay current with emerging AI testing methodologies, tools, and industry best practices.

Requirements

• 6+ years of experience in Quality Assurance, Test Automation, or Quality Engineering.
• 2+ years of hands‑on experience testing Generative AI, LLM‑based applications, or AI‑powered enterprise solutions.
• Strong understanding of Generative AI, Large Language Models (LLMs), Prompt Engineering, and AI application architectures.
• Experience with AI platforms such as Azure OpenAI, OpenAI GPT, Anthropic Claude, Google Gemini, or Meta Llama.
• Hands‑on experience testing AI chatbots, virtual assistants, AI agents, and RAG‑based applications.
• Strong expertise in automation testing using Playwright, Selenium, Cypress, or similar frameworks.
• Proficiency in Python for automation testing and AI evaluation frameworks.
• Experience with Postman, REST Assured, Swagger/OpenAPI, and API validation methodologies.
• Knowledge of AI evaluation frameworks such as Promptfoo, DeepEval, Ragas, LangSmith, TruLens, or similar tools.
• Strong understanding of AI security testing, including prompt injection and adversarial testing techniques.
• Experience working with cloud platforms, preferably Microsoft Azure and Azure AI Services.
• Familiarity with SQL, database validation, and data quality testing.
• Experience with Azure DevOps, GitHub Actions, Jenkins, or GitLab CI/CD.
• Strong analytical, problem‑solving, communication, and stakeholder management skills.

Preferred, but not required

• Experience within Banking, Financial Services, Capital Markets, or Insurance domains.
• Experience testing Agentic AI and Multi‑Agent Systems.
• Familiarity with LangChain, LangGraph, Semantic Kernel, CrewAI, or LlamaIndex.
• Knowledge of vector databases such as Pinecone, Weaviate, ChromaDB, Qdrant, or Azure AI Search.
• Experience with AI observability, monitoring, and governance platforms.
• Exposure to MLOps, model lifecycle management, and AI governance frameworks.
• Azure AI Engineer, Automation Testing, or Cloud certifications.
• Experience establishing enterprise AI testing standards and quality frameworks.

We offer:

• A highly competitive compensation and benefits package.
• A multinational organization with 60 offices in 20 countries and the possibility to work abroad.
• 10 days of paid annual leave (plus sick leave and national holidays).
• Maternity & paternity leave plans.
• A comprehensive insurance plan including medical, dental, vision, life insurance, and long‑/short‑term disability (plans vary by region).
• Retirement savings plans.
• A higher education certification policy.
• Commuter benefits (varies by region).
• Extensive training opportunities, focused on skills, substantive knowledge, and personal development.
• On‑demand Udemy for Business for all Synechron employees with free access to more than 5000 curated courses.
• Coaching opportunities with experienced colleagues from our Financial Innovation Labs (FinLabs) and Center of Excellence (CoE) groups.
• Cutting edge projects at the world’s leading tier‑one banks, financial institutions and insurance firms.
• A flat and approachable organization.
• A truly diverse, fun‑loving, and global work culture.

SYNECHRON’S DIVERSITY & INCLUSION STATEMENT

Diversity & Inclusion are fundamental to our culture, and Synechron is proud to be an equal opportunity workplace and is an affirmative action employer. Our Diversity, Equity, and Inclusion (DEI) initiative ‘Same Difference’ is committed to fostering an inclusive culture – promoting equality, diversity and an environment that is respectful to all. We strongly believe that a diverse workforce helps build stronger, successful businesses as a global company. We encourage applicants from across diverse backgrounds, race, ethnicities, religion, age, marital status, gender, sexual orientations, or disabilities to apply. We empower our global workforce by offering flexible workplace arrangements, mentoring, internal mobility, learning and development programs, and more.

All employment decisions at Synechron are based on business needs, job requirements and individual qualifications, without regard to the applicant’s gender, gender identity, sexual orientation, race, ethnicity, disabled or veteran status, or any other characteristic protected by law.