QA AI Automation Engineer - LLMs & Multi-Agent Quality

Dynasty Financial Partners, LLC

Saint Petersburg (FL)

On-site

USD 120,000 - 150,000

Full time

14 days+

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

Dynasty Financial Partners, LLC is building AI systems for advisors and clients within a regulated wealth management platform. This role focuses on evaluating non-deterministic systems, developing tooling to catch AI regressions, and ensuring code quality in agent-written software.

You will own AI evaluation, design multi-agent QA workflows, and scale automated gates across delivery teams, partnering with AI Labs and product to define coverage and acceptance criteria.

Qualifications

  • 5+ years in software quality, test automation, or SDET with ownership of automation architecture.
  • Production experience with LLM-powered systems: agent patterns, prompt engineering, tool calling, orchestration.
  • Hands-on with model APIs (OpenAI, Anthropic, Azure OpenAI) and AI-assisted development tooling.
  • Strong programming in Python, JavaScript/TypeScript, Java, Kotlin, or C#.
  • Experience building evaluation criteria for systems without a single correct answer.
  • Deep CI/CD fluency: pipeline integration, gating, monitoring, and logging.
  • API testing depth and validating third-party integrations.
  • Clear written communication to explain risks to product owners and engineers.

Responsibilities

  • Own and extend AI evaluation framework for AI chat and API surface.
  • Build multi-agent quality workflow in CI/CD: run, triage, defect writeups, fixes, re-run.
  • Define agent roles, guardrails, and auto-apply vs human review criteria.
  • Make cost-aware decisions on agent vs deterministic automation.
  • Report autonomous resolution rate, false positives, and engineer hours saved.
  • Scale code quality with automated gates for agent-authored code.
  • Partner with product, engineering, and AI Labs on coverage and acceptance.
  • Contribute to release-readiness and go/no-go decisions with evidence.
  • Maintain and modernize automation estate; mentor engineers.

Skills

Automation architecture
LLM experience
CI/CD fluency
API testing
Python
JavaScript/TypeScript
Communication

Tools

Langfuse
Promptfoo
OpenTelemetry
New Relic
n8n
Azure

Job description

Dynasty Financial Partners, LLC is building AI systems for advisors and clients within a regulated wealth management platform. This role focuses on evaluating non-deterministic systems, developing tooling to catch AI regressions, and ensuring code quality in agent-written software.

You will own AI evaluation, design multi-agent QA workflows, and scale automated gates across delivery teams, partnering with AI Labs and product to define coverage and acceptance criteria.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

AI Quality Engineer: Automate Trust in Agent-Powered Systems
AI Quality Engineer: Automate Trust in Agent-Powered Systems

Dynasty Financial Partners • Saint Petersburg (FL)

On-site
USD 120,000 - 150,000
AI Quality Engineer – LLMs, Agents & Automation
AI Quality Engineer – LLMs, Agents & Automation

Newfold Digital • United States

Remote
USD 120,000 - 180,000
Health insurance
401(k) matching
Paid time off
Remote AI QA Engineer for LLMs & Autonomous Agents
Remote AI QA Engineer for LLMs & Autonomous Agents

web • United States

Remote
USD 120,000 - 180,000
QA AI Agent Developer — LLM-Driven Test Automation
QA AI Agent Developer — LLM-Driven Test Automation

Accord Technologies Inc • New Jersey

On-site
USD 90,000 - 120,000
Remote AI Agent Engineer for LLM-Powered QA Automation
Remote AI Agent Engineer for LLM-Powered QA Automation

Itransition • Georgia

Hybrid
USD 120,000 - 190,000
Flexible working hours
Remote options
Medical insurance
+3
Evaluation AI Architect for Multi-Agent LLMs
Evaluation AI Architect for Multi-Agent LLMs

Not Disclosed • United States

Remote
USD 130,000 - 190,000
AI-Native QA Engineer for LLM/Agent Quality
AI-Native QA Engineer for LLM/Agent Quality

Newton Research • Massachusetts

On-site
USD 115,000 - 130,000
Salary equity
LLM & AI QA Engineer: Reliability & Safety Tester
LLM & AI QA Engineer: Reliability & Safety Tester

Synoptek • United States

Remote
USD 90,000 - 120,000
AI QA Engineer for LLM & GenAI Testing & Automation
AI QA Engineer for LLM & GenAI Testing & Automation

Compunnel Inc. • New York (NY)

Hybrid
USD 140,000 - 190,000
Senior AI Agents Engineer — LLM Systems
Senior AI Agents Engineer — LLM Systems

Testing Site - JPMC • New York (NY)

On-site
USD 160,000 - 240,000