Senior AI QA Engineer - Shape Reliable Trading AI

Delta Exchange; Helena (MT); On-site; USD 120,000 - 160,000; Full time

Job summary

Delta Exchange seeks a Senior QA Engineer - AI to own the quality of its AI products, including a customer support chatbot, an API Copilot, and an MCP server. You will define what is correct for probabilistic outputs, build evaluation datasets, and shepherd releases with a rigorous testing regime.

You will design layered scoring, run end-to-end evaluations, and identify failure modes like hallucination and loss of context. This role offers high autonomy and direct impact on product reliability.

Qualifications

• Non-negotiable QA ownership for AI products, including chatbots and agentic systems.
• Hands-on experience building or operating an AI evaluation harness.
• Ability to describe measurements, results, and decisions from harness data.
• 4–6 years of QA/SDET experience.
• Working proficiency in Python for tooling and analysis.
• Strong API testing experience and tooling development ability.

Responsibilities

• Build and maintain golden datasets for AI products and verify against a truth source.
• Design layered scoring: deterministic rules and model-graded evaluation.
• Manage end-to-end evaluation runs, compare against baselines, triage failures.
• Own the release gate for production prompts/models.
• Identify failure modes like hallucination, context loss, and PII exposure.
• Investigate production traces to diagnose root causes.

Tools

Python

Opik

Langfuse

LangSmith

Job description

Delta Exchange seeks a Senior QA Engineer - AI to own the quality of its AI products, including a customer support chatbot, an API Copilot, and an MCP server. You will define what is correct for probabilistic outputs, build evaluation datasets, and shepherd releases with a rigorous testing regime.

You will design layered scoring, run end-to-end evaluations, and identify failure modes like hallucination and loss of context. This role offers high autonomy and direct impact on product reliability.