QA AI Automation Engineer - LLMs & Multi-Agent Quality

Dynasty Financial Partners, LLC; Saint Petersburg (FL); On-site; USD 120,000 - 150,000; Full time

Job summary

Dynasty Financial Partners, LLC is building AI systems for advisors and clients within a regulated wealth management platform. This role focuses on evaluating non-deterministic systems, developing tooling to catch AI regressions, and ensuring code quality in agent-written software.

You will own AI evaluation, design multi-agent QA workflows, and scale automated gates across delivery teams, partnering with AI Labs and product to define coverage and acceptance criteria.

Qualifications

• 5+ years in software quality, test automation, or SDET with ownership of automation architecture.
• Production experience with LLM-powered systems: agent patterns, prompt engineering, tool calling, orchestration.
• Hands-on with model APIs (OpenAI, Anthropic, Azure OpenAI) and AI-assisted development tooling.
• Strong programming in Python, JavaScript/TypeScript, Java, Kotlin, or C#.
• Experience building evaluation criteria for systems without a single correct answer.
• Deep CI/CD fluency: pipeline integration, gating, monitoring, and logging.
• API testing depth and validating third-party integrations.
• Clear written communication to explain risks to product owners and engineers.

Responsibilities

• Own and extend AI evaluation framework for AI chat and API surface.
• Build multi-agent quality workflow in CI/CD: run, triage, defect writeups, fixes, re-run.
• Define agent roles, guardrails, and auto-apply vs human review criteria.
• Make cost-aware decisions on agent vs deterministic automation.
• Report autonomous resolution rate, false positives, and engineer hours saved.
• Scale code quality with automated gates for agent-authored code.
• Partner with product, engineering, and AI Labs on coverage and acceptance.
• Contribute to release-readiness and go/no-go decisions with evidence.
• Maintain and modernize automation estate; mentor engineers.

Skills

Automation architecture

LLM experience

CI/CD fluency

API testing

Python

JavaScript/TypeScript

Communication

Tools

Langfuse

Promptfoo

OpenTelemetry

New Relic

n8n

Azure

Job description

Dynasty Financial Partners, LLC is building AI systems for advisors and clients within a regulated wealth management platform. This role focuses on evaluating non-deterministic systems, developing tooling to catch AI regressions, and ensuring code quality in agent-written software.

You will own AI evaluation, design multi-agent QA workflows, and scale automated gates across delivery teams, partnering with AI Labs and product to define coverage and acceptance criteria.