Engineering Manager, Agent Prompts & Evals

Tactical Edge

Washington (District of Columbia)

On-site

USD 180,000 - 240,000

Full time

14 days+

Get more replies from employers

Send a job-specific resume in minutes.

Benefits offered by this job

Flexible work setup

Job summary

Tactical Edge in Washington, DC is seeking a hands-on engineering manager to lead the team responsible for prompt engineering, model evaluation, and AI quality assurance across our AI products. You will own the systems that ensure our agents produce reliable, accurate, and safe outputs.

This player-coach role sets the technical direction, builds the evaluation infrastructure, and grows a team of specialists while continuing to write code and review prompts hands-on, with a strong focus on LLM

Qualifications

5+ years engineering experience with 2+ years managing teams.
Deep experience with LLM prompting, evaluation, and optimization.
Familiarity with eval frameworks (Braintrust, Langsmith, or custom).
Production AI systems experience with observability and monitoring.
Understanding of model architectures, tokenization, and inference optimization.

Responsibilities

Manage and grow a team of prompt engineers and eval specialists.
Set standards for prompt design, evaluation methodology, and quality metrics across all products.
Design and operate eval pipelines that measure accuracy, safety, hallucination rates, and task completion across all AI agents.
Build automated benchmarks and regression suites.
Run A/B tests, cost-quality tradeoff analysis, and latency benchmarks.
Own the quality bar for AI outputs.
Implement red-teaming, adversarial testing, and safety evaluations.
Build dashboards for monitoring AI quality in production.
Work with product, engineering, and customer teams to translate requirements into evaluation criteria and prompt strategies.

Skills

LLM prompting

Evaluation

Team leadership

Observability

Monitoring

Prompt engineering

Cross-functional collaboration

Tools

Braintrust

Langsmith

Custom eval framework

Job description

We're looking for a hands-on engineering manager to lead the team responsible for prompt engineering, model evaluation, and AI quality assurance across all Tactical Edge products. You'll own the systems that ensure our AI agents produce reliable, accurate, and safe outputs.

This is a player-coach role — you set the technical direction, build the evaluation infrastructure, and hold the quality bar while growing a team of specialists.

Key Responsibilities

Manage and grow a team of prompt engineers and eval specialists.
Set standards for prompt design, evaluation methodology, and quality metrics across all products.

Prompt Engineering at Scale

Build and maintain prompt libraries, templates, and versioning systems.
Establish best practices for system prompts, few-shot examples, chain-of-thought reasoning, and tool-use instructions.

Evaluation Systems

Design and operate eval pipelines that measure accuracy, safety, hallucination rates, and task completion across all AI agents.
Build automated benchmarks and regression suites.
Run A/B tests, cost-quality tradeoff analysis, and latency benchmarks.

Quality & Safety

Own the quality bar for AI outputs.
Implement red-teaming, adversarial testing, and safety evaluations.
Build dashboards for monitoring AI quality in production.

Cross-functional Collaboration

Work with product, engineering, and customer teams to translate requirements into evaluation criteria and prompt strategies.
Technical manager who still writes code and reviews prompts hands-on.
Deep understanding of LLM behavior, failure modes, and edge cases.
Data-driven decision maker who builds systems to measure what matters.
Strong communicator who can translate AI quality concepts for non-technical stakeholders.
High bar for quality with a pragmatic approach to shipping.

Preferred Qualifications