1. AI - QA & Prompt Analyst

Trinityaiboston; Boston (MA); Hybrid; USD 110,000 - 150,000; Full time

Job summary

Trinityaiboston is seeking an AI QA & Prompt Analyst to ensure the quality, reliability, and performance of our AI models and systems. You will design test strategies for LLM-based applications and collaborate with our AI engineering teams.

The role focuses on designing and executing test plans, developing prompts, and creating automated testing frameworks to identify limitations and biases, while working cross-functionally with engineers to boost model performance.

Qualifications

• 3+ years of QA experience, with at least 1 year in AI/ML testing.
• Strong understanding of LLMs and prompt engineering principles.
• Experience with Python for test automation.
• Familiarity with AI evaluation metrics and methodologies.
• Excellent analytical and problem-solving skills.

Responsibilities

• Design and execute test plans for LLM-based applications and AI systems.
• Develop and optimize prompts for various AI use cases.
• Identify and document AI model limitations, biases, and failure modes.
• Create automated testing frameworks for AI quality assurance.
• Collaborate with AI engineers to improve model performance.

Skills

QA experience

AI/ML testing

Test automation with Python

LLMs knowledge

Tools

Python

MLflow

LangChain

Job description

We are looking for an AI QA & Prompt Analyst to ensure the quality, reliability, and performance of our AI models and systems. You will design test strategies for LLM-based applications and work closely with our AI engineering teams.

What You’ll Do

• Design and execute test plans for LLM-based applications and AI systems
• Develop and optimize prompts for various AI use cases
• Identify and document AI model limitations, biases, and failure modes
• Create automated testing frameworks for AI quality assurance
• Collaborate with AI engineers to improve model performance

Must Have Skills

• 3+ years of QA experience with at least 1 year in AI/ML testing
• Strong understanding of LLMs and prompt engineering principles
• Experience with Python for test automation
• Familiarity with AI evaluation metrics and methodologies
• Excellent analytical and problem-solving skills

Nice-to-Have

• Experience with MLflow, LangChain, or similar frameworks
• Knowledge of RAG architectures and vector databases