AI Quality Engineer: Automate Trust in Agent-Powered Systems

Dynasty Financial Partners

Saint Petersburg (FL)

On-site

USD 120,000 - 150,000

Full time

14 days+

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

Dynasty Financial Partners is seeking an experienced QA engineer to own and extend our AI evaluation framework for advisor-facing AI chat and the API surface. You will build and maintain evaluation suites, curate datasets, and design LLM-based judging rubrics to automate detection of regressions.

You will wire evals into CI/CD, instrument production traces, and drive cost-aware decisions about when to apply deterministic automation versus human review.

Qualifications

  • 5+ years in software quality, test automation, or SDET work with ownership of automation architecture.
  • Production experience with LLM-powered systems: agent patterns, prompt engineering, tool calls, orchestration.
  • Hands-on with major model APIs (OpenAI, Anthropic, Azure OpenAI) and AI-assisted tooling.

Responsibilities

  • Own and extend our AI evaluation framework for AI chat.
  • Build and maintain evaluation suite for our AI assistant and its API surface—correctness, grounding, retrieval quality, and behavior.
  • Curate golden datasets and adversarial test sets from real advisor workflows.

Skills

Software QA
Test automation
SDET ownership
LLM systems
Python
TypeScript/JavaScript
Java
Kotlin
C#
CI/CD
API testing

Tools

Langfuse
Promptfoo
OpenTelemetry
New Relic
n8n
Azure

Job description

Dynasty Financial Partners is seeking an experienced QA engineer to own and extend our AI evaluation framework for advisor-facing AI chat and the API surface. You will build and maintain evaluation suites, curate datasets, and design LLM-based judging rubrics to automate detection of regressions.

You will wire evals into CI/CD, instrument production traces, and drive cost-aware decisions about when to apply deterministic automation versus human review.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

QA AI Automation Engineer - LLMs & Multi-Agent Quality
QA AI Automation Engineer - LLMs & Multi-Agent Quality

Dynasty Financial Partners, LLC • Saint Petersburg (FL)

On-site
USD 120,000 - 150,000
AI Quality Engineer – LLMs, Agents & Automation
AI Quality Engineer – LLMs, Agents & Automation

Newfold Digital • United States

Remote
USD 120,000 - 180,000
Health insurance
401(k) matching
Paid time off
QA AI Automation Engineer
QA AI Automation Engineer

Dynasty Financial Partners, LLC • Saint Petersburg (FL)

On-site
USD 120,000 - 150,000
AI Engineer, Agents & Quality - Build Tools & Evaluation
AI Engineer, Agents & Quality - Build Tools & Evaluation

Gold Rose AI • United States

Hybrid
USD 120,000 - 180,000
QA AI Automation Engineer
QA AI Automation Engineer

Dynasty Financial Partners • Saint Petersburg (FL)

On-site
USD 120,000 - 150,000
Remote AI QA Engineer for LLMs & Autonomous Agents
Remote AI QA Engineer for LLMs & Autonomous Agents

web • United States

Remote
USD 120,000 - 180,000
AI Quality Architect: GenAI-Powered QA & Architecture
AI Quality Architect: GenAI-Powered QA & Architecture

EPAM Systems Inc • United States

Remote
USD 140,000 - 210,000
Senior AI-Driven QA Automation Engineer
Senior AI-Driven QA Automation Engineer

Data Innovations • United States

Remote
USD 119,000 - 141,000
Salary range: $119,200 - $141,200
Bonus eligibility for non-sales staff
Benefits: medical, dental, vision,life
Lead AI Evaluation Engineer — Agent Quality
Lead AI Evaluation Engineer — Agent Quality

FurtherAI • San Francisco (CA)

On-site
USD 140,000 - 210,000
Fully covered health, dental, and visa
Competitive compensation, stock + 401k
Unlimited PTO
+4
AI-Native QA Engineer for LLM/Agent Quality
AI-Native QA Engineer for LLM/Agent Quality

Newton Research • Massachusetts

On-site
USD 115,000 - 130,000
Salary equity