Lead AI Engineer – Agentic Test Automation

Alchemy

Tysons (VA)

Hybrid

USD 80,000 - 113,000

Full time

9 days ago

Get more replies from employers

Send a job-specific resume in minutes.

Job summary

Alchemy is seeking a Lead AI Engineer in a hybrid Tysons, VA role focused on agentic test automation powered by LLM-powered agents. You will design and operate multi-step AI agents to automate test generation, validation, and reporting within CI/CD pipelines using GitHub Actions.

You will build agent workflows for test generation, requirements review, and automated reporting across microservices, integrating Karate and Playwright for API and UI testing, with strong emphasis on guardrails and

Qualifications

Hands-on experience building LLM-powered agents (tool-using, multi-step reasoning, guardrails).
Experience with prompting patterns, structured outputs using JSON schemas, evaluation, and reducing hallucinations.
Strong experience designing agent workflows for test generation/augmentation, requirements review/completeness validation, and report generation/summarization.
Strong proficiency with GitHub Copilot in daily development.
Deep experience with GitHub Actions (CI/CD pipelines, reusable workflows, composite actions).
Deep experience with GitHub capabilities including PR checks, branch protections, CODEOWNERS, and templates.

Responsibilities

Build and operate tool-using, multi-step LLM-powered agents with guardrails.
Design agent workflows for test generation and augmentation.
Design agent workflows for requirements review and completeness validation.
Design agent workflows for report generation and summarization.
Implement and evolve CI/CD and test automation frameworks using GitHub Actions and supporting GitHub capabilities.
Use GitHub Copilot in day-to-day development.
Leverage GitHub platform features including PR checks, branch protections, CODEOWNERS, and templates.
Automate workflows via GitHub APIs and webhooks as needed.
Design and implement automation with Karate (API testing, contract-like checks, data-driven testing, mocks).
Design and implement automation with Playwright (UI automation, selectors strategy, parallelization, and trace/video artifacts).
Aggregate and normalize results across multiple microservices and pipelines.
Produce actionable automated reports including trend analysis, failure clustering, and service correlation.

Skills

LLM-powered agents experience

Test automation design

Guardrails in AI workflows

Prompting patterns

Structured outputs JSON schemas

Cross-functional collaboration

Tools

Karate

Playwright

GitHub Actions

GitHub APIs

GitHub webhooks

Job description

Alchemy is hiring for a Lead AI Engineer role focused on agentic test automation powered by LLM-powered agents. This hybrid position is based in Tysons, VA (22182), and you'll lead hands-on work that connects automated test generation, validation, and reporting with modern engineering workflows in GitHub.

You will design agent workflows that help teams produce and maintain reliable tests, ensure requirements are complete, and generate actionable summaries across pipelines and microservices. The work blends agent engineering (tool use, guardrails, structured outputs) with proven test automation frameworks including Karate and Playwright, integrated into CI/CD using GitHub Actions.

What you'll do

Build and operate tool-using, multi-step LLM-powered agents with guardrails.
Design agent workflows for test generation and augmentation.
Design agent workflows for requirements review and completeness validation.
Design agent workflows for report generation and summarization.
Implement and evolve CI/CD and test automation frameworks using GitHub Actions and supporting GitHub capabilities.
Use GitHub Copilot in day-to-day development.
Leverage GitHub platform features including PR checks, branch protections, CODEOWNERS, and templates.
Automate workflows via GitHub APIs and webhooks as needed.
Design and implement automation with Karate (API testing, contract-like checks, data-driven testing, mocks).
Design and implement automation with Playwright (UI automation, selectors strategy, parallelization, and trace/video artifacts).
Aggregate and normalize results across multiple microservices and multiple pipelines.
Produce actionable automated reports including trend analysis, failure clustering, and service correlation.

What you bring

Hands-on experience building LLM-powered agents (tool-using, multi-step reasoning, guardrails).
Experience with prompting patterns, structured outputs using JSON schemas, evaluation, and reducing hallucinations.
Strong experience designing agent workflows for test generation/augmentation, requirements review/completeness validation, and report generation/summarization.
Strong proficiency with GitHub Copilot in daily development.
Deep experience with GitHub Actions (CI/CD pipelines, reusable workflows, composite actions).
Deep experience with GitHub capabilities including PR checks, branch protections, CODEOWNERS, and templates.
Automation experience using GitHub APIs/webhooks as needed.
Advanced automation experience with Karate and Playwright as described above.
Strong understanding of test design and coverage across happy path, negative/validation, and edge/boundary scenarios.
Experience with data setup/teardown strategies and test isolation.
Proven ability to aggregate and normalize results from multiple microservices and multiple pipelines.
Experience producing actionable automated reports including trend analysis, failure clustering, and service correlation.