Senior Prompt Engineer: LLM Performance & Validation

Welo Global; United States; Remote; USD 78,000 - 134,000; Full time

Job summary

Welo Global is seeking a Prompt Engineer to manage end-to-end migration workflows for transitioning templates to LLM autoraters. You will leverage internal tools to maximize model performance through prompt engineering techniques and rigorous evaluation.

You will use APG to generate baseline prompts, supervise APO outputs, and manually refine prompts for complex architectures. Strong data analysis and SQL skills are essential for measuring accuracy against gold data.

Qualifications

• Native fluency in French and fluent English.
• Based in France.
• 4+ years as Prompt Engineer with LLM tuning for structured outputs.

Responsibilities

• Use APG tools to create baseline prompts for template clusters.
• Run APO tool, review outputs, flag deadlocks or plateaus.
• Manually draft and refine prompts for complex templates and edge cases.
• Monitor shadowbot runs and ensure disagreements are tracked.
• Evaluate prompts against gold data and compute F1, precision, recall.

Skills

French (native)

English (fluent)

SQL

Education

Bachelor’s degree in CS/DS/Computational Linguistics

Master’s degree in CS/DS

PhD in related field

Tools

Goose API

Job description

Welo Global is seeking a Prompt Engineer to manage end-to-end migration workflows for transitioning templates to LLM autoraters. You will leverage internal tools to maximize model performance through prompt engineering techniques and rigorous evaluation.

You will use APG to generate baseline prompts, supervise APO outputs, and manually refine prompts for complex architectures. Strong data analysis and SQL skills are essential for measuring accuracy against gold data.