AI Quality Engineer – LLMs, Agents & Automation

Newfold Digital

United States

Remote

USD 120,000 - 180,000

Full time

8 days ago

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Benefits offered by this job

Health insurance
401(k) matching
Paid time off

Job summary

Newfold Digital is seeking an experienced QA engineer focused on AI-powered systems and agentic workflows. The role involves validating, evaluating, and improving AI/LLM-driven applications, building scalable automation, and ensuring reliability across modern AI-native platforms.

You will collaborate with Engineering, AI/ML, Platform, Product, and DevOps teams to implement robust testing strategies, synthetic datasets, and evaluation metrics across multi-agent and RAG pipelines.

Qualifications

  • Hands-on testing of AI/LLM-based systems and agents.
  • Experience validating AI governance, prompt engineering, and evals.
  • Strong debugging, analytical, and troubleshooting skills.

Responsibilities

  • Design and validate testing strategies for AI-powered applications, copilots, chatbots, and LLM-driven workflows.
  • Develop AI evaluation frameworks for hallucination detection, groundedness, toxicity, and safety.
  • Build automated validation pipelines for LLM outputs with semantic scoring and metrics.
  • Integrate automation into CI/CD pipelines and quality gates.
  • Collaborate with Engineering, AI/ML, Platform and DevOps teams to ensure reliability and observability.

Skills

Python
AI/LLM testing
QA principles
Agile/Scrum
Problem solving

Education

Bachelor's degree in CS/Engineering

Tools

Robot Framework
Playwright
GitHub Actions
Docker
Kubernetes

Job description

Newfold Digital is seeking an experienced QA engineer focused on AI-powered systems and agentic workflows. The role involves validating, evaluating, and improving AI/LLM-driven applications, building scalable automation, and ensuring reliability across modern AI-native platforms.

You will collaborate with Engineering, AI/ML, Platform, Product, and DevOps teams to implement robust testing strategies, synthetic datasets, and evaluation metrics across multi-agent and RAG pipelines.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

AI Test Engineer (LLM and AI QA)
AI Test Engineer (LLM and AI QA)

Synoptek • United States

Remote
USD 90,000 - 120,000
Remote AI QA Engineer for LLMs & Autonomous Agents
Remote AI QA Engineer for LLMs & Autonomous Agents

web • United States

Remote
USD 120,000 - 180,000
AI Quality Engineer: LLM Testing & Validation
AI Quality Engineer: LLM Testing & Validation

Global Business Ser. 4u Inc. • Plano (TX)

On-site
USD 120,000 - 180,000
QA AI Automation Engineer - LLMs & Multi-Agent Quality
QA AI Automation Engineer - LLMs & Multi-Agent Quality

Dynasty Financial Partners, LLC • Saint Petersburg (FL)

On-site
USD 120,000 - 150,000
Remote AI Agent Engineer for LLM-Powered QA Automation
Remote AI Agent Engineer for LLM-Powered QA Automation

Itransition • Georgia

Hybrid
USD 120,000 - 190,000
Flexible working hours
Remote options
Medical insurance
+3
AI-Native QA Engineer for LLM/Agent Quality
AI-Native QA Engineer for LLM/Agent Quality

Newton Research • Massachusetts

On-site
USD 115,000 - 130,000
Salary equity
QA / Automation Engineer Agentic AI
QA / Automation Engineer Agentic AI

Compunnel Inc. • Atlanta (GA)

On-site
USD 110,000 - 160,000
QA Automation Engineer
QA Automation Engineer

GCS Recruitment • United States

Remote
USD 120,000 - 170,000
AI QA Engineer
AI QA Engineer

Cavendish Professionals • Town of Italy (NY)

On-site
USD 95,000 - 120,000
AI Quality Assurance Engineer - ML/RPA Testing
AI Quality Assurance Engineer - ML/RPA Testing

Quantum Technologies. LLC • Dallas (TX)

On-site
USD 110,000 - 150,000