AI Test Engineer (LLM and AI QA)

Synoptek; United States; À distance; USD 90 000 - 120 000; Plein temps

Résumé du poste

Synoptek is seeking an AI Tester to evaluate the quality, safety, and reliability of AI-powered products, including LLM applications, agents, and ML-driven features. You will design test strategies for non-deterministic systems and help ensure our AI behaves accurately, safely, and consistently in production.

Design and execute test plans for LLM, RAG, and agentic workflows; build evaluation datasets with golden sets and adversarial prompts; assess output quality, and perform red-teaming.

Qualifications

• 4+ years in QA, test automation, or data/ML evaluation.
• Solid understanding of how LLMs work (prompts, context windows, temperature, embeddings).
• Experience testing APIs and writing automated tests (Python preferred).
• Strong analytical skills and attention to detail.
• Clear written communication for documenting non-deterministic behavior.

Responsabilités

• Design and execute test plans for LLM, RAG, and agentic workflows.
• Build evaluation datasets (golden sets, edge cases, adversarial prompts).
• Assess output quality: accuracy, relevance, hallucination, tone, and consistency.
• Perform red-teaming: prompt injection, jailbreaks, data leakage, and bias testing.
• Validate tool-calling, multi-step agent behavior, and guardrails
• Automate regression and evaluation pipelines (e.g., LLM-as-judge, scoring harnesses).
• Track metrics such as latency, cost per request, and failure rates
• Log, triage, and reproduce defects with clear prompts, inputs, and model versions
• Partner with ML engineers, product owners, and developers to improve prompts and models
• Document findings and contribute to release go/no-go decisions

Connaissances

QA expertise

Test automation

Data/ML evaluation

Analytical thinking

Clear written communication

Outils

Python

APIs testing

LangSmith

Ragas

DeepEval

Description du poste

**About the Role**

We're looking for an AI Tester to evaluate the quality, safety, and reliability of AI-powered products, including LLM applications, agents, and ML-driven features. You'll design test strategies for non-deterministic systems and help ensure our AI behaves accurately, safely, and consistently in production.

**Responsibilities**

• Design and execute test plans for LLM, RAG, and agentic workflows
• Build evaluation datasets (golden sets, edge cases, adversarial prompts)
• Assess output quality: accuracy, relevance, hallucination, tone, and consistency
• Perform red-teaming: prompt injection, jailbreaks, data leakage, and bias testing
• Validate tool-calling, multi-step agent behavior, and guardrails
• Automate regression and evaluation pipelines (e.g., LLM-as-judge, scoring harnesses)
• Track metrics such as latency, cost per request, and failure rates
• Log, triage, and reproduce defects with clear prompts, inputs, and model versions
• Partner with ML engineers, product owners, and developers to improve prompts and models
• Document findings and contribute to release go/no-go decisions

**Required Qualifications**

• 4+ years in QA, test automation, or data/ML evaluation
• Solid understanding of how LLMs work (prompts, context windows, temperature, embeddings)
• Experience testing APIs and writing automated tests (Python preferred)
• Strong analytical skills and attention to detail
• Clear written communication for documenting non-deterministic behavior

**Preferred Qualifications**

• Experience with evaluation tools (e.g.,LangSmith, Ragas, DeepEval)
• Familiarity with RAG pipelines, vector databases, and agent frameworks
• Knowledge of AI safety, responsible AI, and compliance practices
• Cloud platform experience (Azure)
• Background in test automation frameworks