LLM & AI QA Engineer: Reliability & Safety Tester

Synoptek; United States; Remote; USD 90,000 - 120,000; Full time

Job summary

Synoptek is seeking an AI Tester to evaluate the quality, safety, and reliability of AI-powered products, including LLM applications, agents, and ML-driven features. You will design test strategies for non-deterministic systems and help ensure our AI behaves accurately, safely, and consistently in production.

Design and execute test plans for LLM, RAG, and agentic workflows; build evaluation datasets with golden sets and adversarial prompts; assess output quality, and perform red-teaming.

Qualifications

• 4+ years in QA, test automation, or data/ML evaluation.
• Solid understanding of how LLMs work (prompts, context windows, temperature, embeddings).
• Experience testing APIs and writing automated tests (Python preferred).
• Strong analytical skills and attention to detail.
• Clear written communication for documenting non-deterministic behavior.

Responsibilities

• Design and execute test plans for LLM, RAG, and agentic workflows.
• Build evaluation datasets (golden sets, edge cases, adversarial prompts).
• Assess output quality: accuracy, relevance, hallucination, tone, and consistency.
• Perform red-teaming: prompt injection, jailbreaks, data leakage, and bias testing.
• Validate tool-calling, multi-step agent behavior, and guardrails
• Automate regression and evaluation pipelines (e.g., LLM-as-judge, scoring harnesses).
• Track metrics such as latency, cost per request, and failure rates
• Log, triage, and reproduce defects with clear prompts, inputs, and model versions
• Partner with ML engineers, product owners, and developers to improve prompts and models
• Document findings and contribute to release go/no-go decisions

Skills

QA expertise

Test automation

Data/ML evaluation

Analytical thinking

Clear written communication

Tools

Python

APIs testing

LangSmith

Ragas

DeepEval

Job description

Synoptek is seeking an AI Tester to evaluate the quality, safety, and reliability of AI-powered products, including LLM applications, agents, and ML-driven features. You will design test strategies for non-deterministic systems and help ensure our AI behaves accurately, safely, and consistently in production.

Design and execute test plans for LLM, RAG, and agentic workflows; build evaluation datasets with golden sets and adversarial prompts; assess output quality, and perform red-teaming.