Lead QA Engineer for AI & LLM Workflows (Remote)

vibehackers; United States; Hybrid; USD 150,000 - 190,000; Full time

Benefits offered by this job

Generous annual bonus opportunity

401(k) with employer match

Medical Insurance

Dental Insurance

Health Savings Account with employer &

Job summary

Vibehackers is seeking a Lead Software Engineer in Test for Agentic AI to drive quality engineering across UI, API, backend, and AI layers. You will design scalable automation and validate LLM workflows, RAG pipelines, and distributed orchestration in cloud-native environments.

You will mentor QA engineers, shape testing strategy, and collaborate with product, software, and DevOps teams to ensure resilient, high‑quality AI‑driven applications. Fully remote in the US.

Qualifications

• Bachelor's degree in Computer Science, MIS, Engineering, or related technical field.
• 8+ years of experience in Software Development or SDET/SET roles.
• Strong experience with automation tools such as Playwright, Cypress, Postman, K6, or equivalent frameworks.
• 2+ years testing AI-powered applications, LLM systems, Agentic AI platforms, orchestration frameworks, or autonomous workflows.
• Hands-on experience validating agent orchestration workflows, conversational AI applications, multi-step reasoning systems, and AI-driven business processes.
• Experience testing AI orchestration frameworks such as LangGraph, CrewAI, AutoGen, Semantic Kernel, or similar technologies.
• Expertise in API testing, backend integration testing, UI automation, and end-to-end workflow validation.
• Solid understanding of automation architecture, design patterns, dependency injection, and inversion of control.
• Experience with cloud-native and containerized environments (Azure, AWS, GCP, Docker, Kubernetes) and integrating automated tests into CI/CD pipelines.
• Experience validating asynchronous, event-driven, and distributed systems with emphasis on resiliency and fault tolerance.
• Strong analytical, debugging, troubleshooting, and problem-solving skills.
• Excellent organizational, communication, and stakeholder collaboration skills in Agile environments.
• Experience with AI observability, telemetry, RAG pipeline validation, vector databases, or prompt evaluation frameworks is preferred.

Responsibilities

• Design, develop, and maintain scalable automation frameworks and regression test suites for UI, API, backend, and AI-driven applications.
• Lead testing initiatives for Agentic AI systems including LLM workflows, RAG pipelines, autonomous agents, and distributed orchestration platforms.
• Validate complex agent behaviors such as multi-agent coordination, tool calling, prompt chaining, memory retention, conversational workflows, and fallback/retry mechanisms.
• Develop and execute functional, integration, regression, end-to-end, and performance testing strategies in cloud-native and distributed environments.
• Collaborate with Software Engineers, Product Managers, and DevOps to integrate testing throughout the SDLC and CI/CD pipelines.
• Build reusable automation solutions for APIs, orchestration layers, asynchronous workflows, enterprise integrations, and AI-driven business processes.
• Design and maintain test environments, CI/CD integrations, cloud execution platforms, and automation pipelines to improve efficiency and reliability.
• Establish best practices for code quality, maintainability, scalability, dependency injection, observability, and testability.
• Lead validation of AI workflow resiliency, distributed system reliability, data consistency, and orchestration performance under failure and edge-case conditions.
• Mentor QA engineers and develop scalable QA standards, automation strategies, reporting practices, and continuous improvement initiatives.

Job description

Lead Software Engineer in Test – Agentic AI (Remote - US)

Focuses on testing and validating agentic AI, LLM workflows and RAG pipelines—so it's about building and ensuring AI-driven systems work; leans into rapid AI iteration and AI dev tooling.

About the Role

Lead quality engineering for agentic AI products, designing scalable automation frameworks and validating complex LLM workflows, RAG pipelines, and distributed orchestration systems across UI, API, backend, and AI layers. Drive testing strategy, CI/CD integration, and mentor QA engineers to ensure resilient, performant AI-driven applications.

Job Description

Role

The Lead Software Engineer in Test – Agentic AI will lead quality engineering strategy and execution for AI-powered applications and distributed systems. The role focuses on designing scalable automation frameworks, validating agentic AI behaviors, and ensuring high-quality delivery across UI, API, backend, orchestration, and cloud-native layers.

Key Responsibilities

• Design, develop, and maintain scalable automation frameworks and regression test suites for UI, API, backend, and AI-driven applications.
• Lead testing initiatives for Agentic AI systems including LLM workflows, RAG pipelines, autonomous agents, and distributed orchestration platforms.
• Validate complex agent behaviors such as multi-agent coordination, tool calling, prompt chaining, memory retention, conversational workflows, and fallback/retry mechanisms.
• Develop and execute functional, integration, regression, end-to-end, and performance testing strategies in cloud-native and distributed environments.
• Collaborate with Software Engineers, Product Managers, and DevOps to integrate testing throughout the SDLC and CI/CD pipelines.
• Build reusable automation solutions for APIs, orchestration layers, asynchronous workflows, enterprise integrations, and AI-driven business processes.
• Design and maintain test environments, CI/CD integrations, cloud execution platforms, and automation pipelines to improve efficiency and reliability.
• Establish best practices for code quality, maintainability, scalability, dependency injection, observability, and testability.
• Lead validation of AI workflow resiliency, distributed system reliability, data consistency, and orchestration performance under failure and edge-case conditions.
• Mentor QA engineers and develop scalable QA standards, automation strategies, reporting practices, and continuous improvement initiatives.

Requirements

• Bachelor’s degree in Computer Science, MIS, Engineering, or a related technical field.
• 8+ years of experience in Software Development or SDET/SET roles.
• Strong experience with automation tools such as Playwright, Cypress, Postman, K6, or equivalent frameworks.
• 2+ years testing AI-powered applications, LLM systems, Agentic AI platforms, orchestration frameworks, or autonomous workflows.
• Hands-on experience validating agent orchestration workflows, conversational AI applications, multi-step reasoning systems, and AI-driven business processes.
• Experience testing AI orchestration frameworks such as LangGraph, CrewAI, AutoGen, Semantic Kernel, or similar technologies.
• Expertise in API testing, backend integration testing, UI automation, and end-to-end workflow validation.
• Solid understanding of automation architecture, design patterns, dependency injection, and inversion of control.
• Experience with cloud-native and containerized environments (Azure, AWS, GCP, Docker, Kubernetes) and integrating automated tests into CI/CD pipelines.
• Experience validating asynchronous, event-driven, and distributed systems with emphasis on resiliency and fault tolerance.
• Strong analytical, debugging, troubleshooting, and problem-solving skills.
• Excellent organizational, communication, and stakeholder collaboration skills in Agile environments.
• Experience with AI observability, telemetry, RAG pipeline validation, vector databases, or prompt evaluation frameworks is preferred.

Preferred Qualifications

• Experience with Azure DevOps, GitHub Actions, or similar CI/CD platforms.
• Experience validating RAG pipelines, vector databases, and AI response accuracy.
• Knowledge of healthcare systems, HIPAA compliance, and regulated enterprise environments.
• Experience with event-driven architectures, messaging systems, and distributed data platforms (including SQL Server).
• Prior experience leading QA teams, mentoring engineers, and implementing enterprise test reporting and quality metrics.
• Generous annual bonus opportunity
• 401(k) with employer match
• Flexible Time Off and 11 paid holidays
• LinkedIn Learning access
• Medical, Dental, and Vision plans
• Health Savings Account with employer match and Flexible Spending Account
• 100% company-paid parental leave (after 6 months)
• Company-paid life insurance and short/long-term disability
• Peer recognition program and Employee Assistance Program (free counseling)
• Corporate discounts on retail, travel, and entertainment
• Primarily computer-based work

Compliance

All activities must comply with EEO laws, HIPAA, ERISA, and other applicable regulations.

Skills

Test Automation Quality Engineering Automation Framework Design AI Testing Agentic AI Validation API Testing UI Automation Integration Testing End-to-End Testing Performance Testing CI/CD Integration Test Environment Management Observability Debugging Troubleshooting Mentoring Stakeholder Communication Agile Cloud-native Practices Resiliency and Fault Tolerance

Experience Level

Experience Level

Senior

Employment Type

Full Time

• • Fully Remote
• • Generous annual bonus opportunity
• • 401(k) with Employer Match
• • LinkedIn Learning access
• • Medical Insurance
• • Dental Insurance
• • Health Savings Account with employer match
• • 100% Company-Paid Parental leave (after 6 months)
• • Short/Long Term Disability Insurance
• • Peer Recognition Program
• • Employee Assistance Program (free counseling)