Lead AI Engineer: Agentic Test Automation (Hybrid)

Alchemy; Tysons (VA); Hybrid; USD 80,000 - 113,000; Full time

Job summary

Alchemy is seeking a Lead AI Engineer in a hybrid Tysons, VA role focused on agentic test automation powered by LLM-powered agents. You will design and operate multi-step AI agents to automate test generation, validation, and reporting within CI/CD pipelines using GitHub Actions.

You will build agent workflows for test generation, requirements review, and automated reporting across microservices, integrating Karate and Playwright for API and UI testing, with strong emphasis on guardrails and

Qualifications

• Hands-on experience building LLM-powered agents (tool-using, multi-step reasoning, guardrails).
• Experience with prompting patterns, structured outputs using JSON schemas, evaluation, and reducing hallucinations.
• Strong experience designing agent workflows for test generation/augmentation, requirements review/completeness validation, and report generation/summarization.
• Strong proficiency with GitHub Copilot in daily development.
• Deep experience with GitHub Actions (CI/CD pipelines, reusable workflows, composite actions).
• Deep experience with GitHub capabilities including PR checks, branch protections, CODEOWNERS, and templates.

Responsibilities

• Build and operate tool-using, multi-step LLM-powered agents with guardrails.
• Design agent workflows for test generation and augmentation.
• Design agent workflows for requirements review and completeness validation.
• Design agent workflows for report generation and summarization.
• Implement and evolve CI/CD and test automation frameworks using GitHub Actions and supporting GitHub capabilities.
• Use GitHub Copilot in day-to-day development.
• Leverage GitHub platform features including PR checks, branch protections, CODEOWNERS, and templates.
• Automate workflows via GitHub APIs and webhooks as needed.
• Design and implement automation with Karate (API testing, contract-like checks, data-driven testing, mocks).
• Design and implement automation with Playwright (UI automation, selectors strategy, parallelization, and trace/video artifacts).
• Aggregate and normalize results across multiple microservices and pipelines.
• Produce actionable automated reports including trend analysis, failure clustering, and service correlation.

Skills

LLM-powered agents experience

Test automation design

Guardrails in AI workflows

Prompting patterns

Structured outputs JSON schemas

Cross-functional collaboration

Tools

Karate

Playwright

GitHub Actions

GitHub APIs

GitHub webhooks

Job description

Alchemy is seeking a Lead AI Engineer in a hybrid Tysons, VA role focused on agentic test automation powered by LLM-powered agents. You will design and operate multi-step AI agents to automate test generation, validation, and reporting within CI/CD pipelines using GitHub Actions.

You will build agent workflows for test generation, requirements review, and automated reporting across microservices, integrating Karate and Playwright for API and UI testing, with strong emphasis on guardrails and