Lead Prompt Engineer & AI Strategy Lead

Remote; San Francisco (CA); Remote; USD 180,000 - 270,000; Full time

Job summary

Remote is seeking a Lead Prompt Engineer to guide a team through migrating templates to LLM autoraters and optimize model performance. You will mentor engineers, shape strategy, and drive AI-driven solutions.

You will supervise APG/APO tooling, design manual prompts for edge cases, and monitor quality metrics to ensure alignment with human baselines. Federal compliance and verification steps are part of the onboarding process.

Qualifications

• Master’s or Doctorate degree in CS, Data Science, Computational Linguistics, HCI, Cognitive Science, or related analytics field.
• 7+ years’ experience as Prompt Engineer with LLM tuning for structured outputs.
• Proficiency with SQL or other data analytics tools for error pattern analysis.
• Ability to quickly learn and master proprietary tools with minimal supervision.
• Excellent verbal and written communication skills.

Responsibilities

• Utilize APG tools to create baseline prompts for complex template clusters.
• Run and supervise APO tool, review outputs, and flag deadlocks or plateaus.
• Draft and refine prompts to handle edge cases where tooling is lacking or broken.
• Monitor shadowbot runs to ensure disagreements between human and LLM ratings are captured.
• Measure autorater quality against human baselines using metrics like F1, precision and recall.
• Draft Launch Certification Documentation for technical launch readiness.

Skills

Prompt Engineer experience

LLM tuning

SQL/data analytics

Proprietary tools

Communication

Learning new tools

Education

Master’s or Doctorate in CS/DS/CL/HCI/Cognitive Science or related

Tools

Goose API

Job description

Remote is seeking a Lead Prompt Engineer to guide a team through migrating templates to LLM autoraters and optimize model performance. You will mentor engineers, shape strategy, and drive AI-driven solutions.

You will supervise APG/APO tooling, design manual prompts for edge cases, and monitor quality metrics to ensure alignment with human baselines. Federal compliance and verification steps are part of the onboarding process.