Lead QA Engineer for AI & LLM Workflows (Remote)

Vibehackers; Northern (KY); Hybrid; USD 150,000 - 190,000; Full time

Benefits offered by this job

Generous annual bonus opportunity

401(k) with employer match

Medical Insurance

Dental Insurance

Health Savings Account with employer &

Job summary

Vibehackers is seeking a Lead Software Engineer in Test for Agentic AI to drive quality engineering across UI, API, backend, and AI layers. You will design scalable automation and validate LLM workflows, RAG pipelines, and distributed orchestration in cloud-native environments.

You will mentor QA engineers, shape testing strategy, and collaborate with product, software, and DevOps teams to ensure resilient, high‑quality AI‑driven applications. Fully remote in the US.

Qualifications

• Bachelor's degree in Computer Science, MIS, Engineering, or related technical field.
• 8+ years of experience in Software Development or SDET/SET roles.
• Strong experience with automation tools such as Playwright, Cypress, Postman, K6, or equivalent frameworks.
• 2+ years testing AI-powered applications, LLM systems, Agentic AI platforms, orchestration frameworks, or autonomous workflows.
• Hands-on experience validating agent orchestration workflows, conversational AI applications, multi-step reasoning systems, and AI-driven business processes.
• Experience testing AI orchestration frameworks such as LangGraph, CrewAI, AutoGen, Semantic Kernel, or similar technologies.
• Expertise in API testing, backend integration testing, UI automation, and end-to-end workflow validation.
• Solid understanding of automation architecture, design patterns, dependency injection, and inversion of control.
• Experience with cloud-native and containerized environments (Azure, AWS, GCP, Docker, Kubernetes) and integrating automated tests into CI/CD pipelines.
• Experience validating asynchronous, event-driven, and distributed systems with emphasis on resiliency and fault tolerance.
• Strong analytical, debugging, troubleshooting, and problem-solving skills.
• Excellent organizational, communication, and stakeholder collaboration skills in Agile environments.
• Experience with AI observability, telemetry, RAG pipeline validation, vector databases, or prompt evaluation frameworks is preferred.

Responsibilities

• Design, develop, and maintain scalable automation frameworks and regression test suites for UI, API, backend, and AI-driven applications.
• Lead testing initiatives for Agentic AI systems including LLM workflows, RAG pipelines, autonomous agents, and distributed orchestration platforms.
• Validate complex agent behaviors such as multi-agent coordination, tool calling, prompt chaining, memory retention, conversational workflows, and fallback/retry mechanisms.
• Develop and execute functional, integration, regression, end-to-end, and performance testing strategies in cloud-native and distributed environments.
• Collaborate with Software Engineers, Product Managers, and DevOps to integrate testing throughout the SDLC and CI/CD pipelines.
• Build reusable automation solutions for APIs, orchestration layers, asynchronous workflows, enterprise integrations, and AI-driven business processes.
• Design and maintain test environments, CI/CD integrations, cloud execution platforms, and automation pipelines to improve efficiency and reliability.
• Establish best practices for code quality, maintainability, scalability, dependency injection, observability, and testability.
• Lead validation of AI workflow resiliency, distributed system reliability, data consistency, and orchestration performance under failure and edge-case conditions.
• Mentor QA engineers and develop scalable QA standards, automation strategies, reporting practices, and continuous improvement initiatives.

Job description

Vibehackers is seeking a Lead Software Engineer in Test for Agentic AI to drive quality engineering across UI, API, backend, and AI layers. You will design scalable automation and validate LLM workflows, RAG pipelines, and distributed orchestration in cloud-native environments.

You will mentor QA engineers, shape testing strategy, and collaborate with product, software, and DevOps teams to ensure resilient, high‑quality AI‑driven applications. Fully remote in the US.