Engineering Manager, Prompting & AI Evaluation

Tactical Edge; Washington (District of Columbia); On-site; USD 180,000 - 240,000; Full time

Benefits offered by this job

Flexible work setup

Job summary

Tactical Edge in Washington, DC is seeking a hands-on engineering manager to lead the team responsible for prompt engineering, model evaluation, and AI quality assurance across our AI products. You will own the systems that ensure our agents produce reliable, accurate, and safe outputs.

This player-coach role sets the technical direction, builds the evaluation infrastructure, and grows a team of specialists while continuing to write code and review prompts hands-on, with a strong focus on LLM

Qualifications

• 5+ years engineering experience with 2+ years managing teams.
• Deep experience with LLM prompting, evaluation, and optimization.
• Familiarity with eval frameworks (Braintrust, Langsmith, or custom).
• Production AI systems experience with observability and monitoring.
• Understanding of model architectures, tokenization, and inference optimization.

Responsibilities

• Manage and grow a team of prompt engineers and eval specialists.
• Set standards for prompt design, evaluation methodology, and quality metrics across all products.
• Design and operate eval pipelines that measure accuracy, safety, hallucination rates, and task completion across all AI agents.
• Build automated benchmarks and regression suites.
• Run A/B tests, cost-quality tradeoff analysis, and latency benchmarks.
• Own the quality bar for AI outputs.
• Implement red-teaming, adversarial testing, and safety evaluations.
• Build dashboards for monitoring AI quality in production.
• Work with product, engineering, and customer teams to translate requirements into evaluation criteria and prompt strategies.

Skills

LLM prompting

Evaluation

Team leadership

Observability

Monitoring

Prompt engineering

Cross-functional collaboration

Tools

Braintrust

Langsmith

Custom eval framework

Job description

Tactical Edge in Washington, DC is seeking a hands-on engineering manager to lead the team responsible for prompt engineering, model evaluation, and AI quality assurance across our AI products. You will own the systems that ensure our agents produce reliable, accurate, and safe outputs.

This player-coach role sets the technical direction, builds the evaluation infrastructure, and grows a team of specialists while continuing to write code and review prompts hands-on, with a strong focus on LLM