AI Quality Engineer: Automate Trust in Agent-Powered Systems

Dynasty Financial Partners; Saint Petersburg (FL); On-site; USD 120,000 - 150,000; Full time

Job summary

Dynasty Financial Partners is seeking an experienced QA engineer to own and extend our AI evaluation framework for advisor-facing AI chat and the API surface. You will build and maintain evaluation suites, curate datasets, and design LLM-based judging rubrics to automate detection of regressions.

You will wire evals into CI/CD, instrument production traces, and drive cost-aware decisions about when to apply deterministic automation versus human review.

Qualifications

• 5+ years in software quality, test automation, or SDET work with ownership of automation architecture.
• Production experience with LLM-powered systems: agent patterns, prompt engineering, tool calls, orchestration.
• Hands-on with major model APIs (OpenAI, Anthropic, Azure OpenAI) and AI-assisted tooling.

Responsibilities

• Own and extend our AI evaluation framework for AI chat.
• Build and maintain evaluation suite for our AI assistant and its API surface—correctness, grounding, retrieval quality, and behavior.
• Curate golden datasets and adversarial test sets from real advisor workflows.

Skills

Software QA

Test automation

SDET ownership

LLM systems

Python

TypeScript/JavaScript

Java

Kotlin

C#

CI/CD

API testing

Tools

Langfuse

Promptfoo

OpenTelemetry

New Relic

n8n

Azure

Job description

Dynasty Financial Partners is seeking an experienced QA engineer to own and extend our AI evaluation framework for advisor-facing AI chat and the API surface. You will build and maintain evaluation suites, curate datasets, and design LLM-based judging rubrics to automate detection of regressions.

You will wire evals into CI/CD, instrument production traces, and drive cost-aware decisions about when to apply deterministic automation versus human review.