Engineering Manager, Agent Prompts & Evals

Tactical Edge; Washington (District of Columbia); Vor Ort; USD 180.000 - 240.000; Vollzeit

Benefits dieser Stelle

Flexible work setup

Zusammenfassung

Tactical Edge in Washington, DC is seeking a hands-on engineering manager to lead the team responsible for prompt engineering, model evaluation, and AI quality assurance across our AI products. You will own the systems that ensure our agents produce reliable, accurate, and safe outputs.

This player-coach role sets the technical direction, builds the evaluation infrastructure, and grows a team of specialists while continuing to write code and review prompts hands-on, with a strong focus on LLM

Qualifikationen

• 5+ years engineering experience with 2+ years managing teams.
• Deep experience with LLM prompting, evaluation, and optimization.
• Familiarity with eval frameworks (Braintrust, Langsmith, or custom).
• Production AI systems experience with observability and monitoring.
• Understanding of model architectures, tokenization, and inference optimization.

Aufgaben

• Manage and grow a team of prompt engineers and eval specialists.
• Set standards for prompt design, evaluation methodology, and quality metrics across all products.
• Design and operate eval pipelines that measure accuracy, safety, hallucination rates, and task completion across all AI agents.
• Build automated benchmarks and regression suites.
• Run A/B tests, cost-quality tradeoff analysis, and latency benchmarks.
• Own the quality bar for AI outputs.
• Implement red-teaming, adversarial testing, and safety evaluations.
• Build dashboards for monitoring AI quality in production.
• Work with product, engineering, and customer teams to translate requirements into evaluation criteria and prompt strategies.

Kenntnisse

LLM prompting

Evaluation

Team leadership

Observability

Monitoring

Prompt engineering

Cross-functional collaboration

Tools

Braintrust

Langsmith

Custom eval framework

Jobbeschreibung

We're looking for a hands-on engineering manager to lead the team responsible for prompt engineering, model evaluation, and AI quality assurance across all Tactical Edge products. You'll own the systems that ensure our AI agents produce reliable, accurate, and safe outputs.

This is a player-coach role — you set the technical direction, build the evaluation infrastructure, and hold the quality bar while growing a team of specialists.

Key Responsibilities

• Manage and grow a team of prompt engineers and eval specialists.
• Set standards for prompt design, evaluation methodology, and quality metrics across all products.

Prompt Engineering at Scale

• Build and maintain prompt libraries, templates, and versioning systems.
• Establish best practices for system prompts, few-shot examples, chain-of-thought reasoning, and tool-use instructions.

Evaluation Systems

• Design and operate eval pipelines that measure accuracy, safety, hallucination rates, and task completion across all AI agents.
• Build automated benchmarks and regression suites.
• Run A/B tests, cost-quality tradeoff analysis, and latency benchmarks.

Quality & Safety

• Own the quality bar for AI outputs.
• Implement red-teaming, adversarial testing, and safety evaluations.
• Build dashboards for monitoring AI quality in production.

Cross-functional Collaboration

• Work with product, engineering, and customer teams to translate requirements into evaluation criteria and prompt strategies.
• Technical manager who still writes code and reviews prompts hands-on.
• Deep understanding of LLM behavior, failure modes, and edge cases.
• Data-driven decision maker who builds systems to measure what matters.
• Strong communicator who can translate AI quality concepts for non-technical stakeholders.
• High bar for quality with a pragmatic approach to shipping.

Preferred Qualifications

• 5+ years engineering experience with 2+ years managing teams.
• Deep experience with LLM prompting, evaluation, and optimization.
• Familiarity with eval frameworks (Braintrust, Langsmith, or custom).
• Production AI systems experience with observability and monitoring.
• Understanding of model architectures, tokenization, and inference optimization.

How We Work

Outcome-driven

Enterprise-first

Agentic by design

Systems that reason and act safely

Small teams, high ownership

Autonomy with accountability

What You'll Get

• Work on real, production AI deployments
• Enterprise-scale challenges and measurable impact
• Cross-functional collaboration and high ownership
• Flexible work setup where applicable

Hiring Process

1. Intro call

   Fit + context

2. Technical discussion

   Prompt engineering, eval design, model selection

3. Leadership & systems interview

   Team management, cross-functional collaboration

4. Final conversation

   Alignment + next steps

We value clarity, ownership, and thoughtful execution over buzzwords.