Senior Prompt Engineer - LLM Evaluation & AI Role

Welo Global

United States

Remote

USD 78,000 - 112,000

Full time

11 days ago

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

Welo Global is seeking a Prompt Engineer based in Germany to lead the end-to-end migration of templates to LLM autoraters. You will use client internal tools and advanced prompting techniques to maximize model performance in a vendor setting.

The role requires 4+ years in prompt engineering, native German and fluent English, and a relevant CS/DS degree. You will craft, test, and refine prompts, monitor APO and shadowbot outputs, and document launches for approval.

Qualifications

  • Must have 4+ years as a Prompt Engineer.
  • Experience tuning LLMs for structured outputs and classification.
  • Familiarity with chain-of-thought and few-shot learning.
  • Strong data analysis skills using SQL or similar tools.

Responsibilities

  • Operate end-to-end migration workflow for templates to LLM autoraters.
  • Use client tools to maximize model performance via prompt engineering.
  • Create, test, and refine prompts for complex template clusters.
  • Monitor and manage Automated Prompt Optimization outputs.
  • Analyze model errors and track metrics like F1, precision, recall.
  • Prepare Launch Certification Documentation for deployment.

Skills

German (native)
English (fluent)
Prompt engineering
LLM tuning
SQL analytics
Communication

Education

Bachelor’s/Master’s/PhD in CS/DS/Computational Linguistics/HCI/Cognitive Science

Tools

Goose API
APO systems
SQL tooling

Job description

Prompt Engineering is one of the fastest-growing niches in AI, and this role sits at a technically sophisticated end of the spectrum — focused on autorater systems, model evaluation metrics (F1, precision, recall), and optimization workflows rather than simple chatbot tuning. That's a meaningful differentiator on a CV.

Skills developed here are in high demand:

  • LLM evaluation methodology (gold data benchmarking, shadowbot monitoring)
  • Automated Prompt Optimization (APO) — still rare expertise in the market
  • Chain-of-thought and few-shot prompting for structured classification
  • SQL-based model performance analysis

Compensation concern: No salary range is listed, which is unusual for a senior role (4+ years experience required). In Germany, senior Prompt Engineers or AI specialists typically earn €70,000–€100,000+ annually. Candidates should probe this in interviews.

Culture signals: The mention of "client's internal tools" and migration workflows suggests this is a vendor/contractor role supporting a larger tech company rather than an in-house position. This may limit long-term career growth within the organization but can build highly specialized, portable expertise.

Green flags: Genuine technical depth; exposure to proprietary enterprise LLM infrastructure; dual-language (German/English) requirement adds scarcity value. Watch out for: No benefits listed, no salary transparency, and the contractor framing — clarify employment type and stability before accepting.

We are seeking a Prompt Engineer to be responsible for the end-to-end technical migration workflow for transitioning templates to LLM autoraters. The role requires using client's internal tools to leverage prompt engineering techniques to maximize model performance.

Responsibilities
  • Utilize Automatic Prompt Generation (APG) tools to create baseline prompts for complex parent-child template clusters.
  • Run and supervise Automated Prompt Optimization (APO) tool, review the outputs, and flag when the APO reaches deadlocks or plateaus.
  • Manually draft, test, and refine prompts to navigate complex template architectures, overcome anti-patterns, and handle edge cases where tooling is lacking or broken. Solve edge-case scenarios by designing and refining manual prompts.
  • Monitor shadowbot runs to ensure sufficient disagreements (between human and LLM ratings) are registered, generated, and tracked.
  • Run prompt versions against established gold data to continuously measure autorater quality against the human crowd baseline, calculating accuracy metrics such as F1 scores, precision, and recall.
  • Draft technical launch readiness justifications (Launch Certification Documentation) for final approval.
Requirements
  • Language Skills: Native fluency in German and fluent in English.
  • Location: Must be based in Germany.
  • Education: Bachelor's, Master's, or Doctorate degree in Computer Science, Data Science, Computational Linguistics, Human-Computer Interaction (HCI), Cognitive Science, or a related analytical field.
  • Prompt Engineering & AI Expertise: At least 4 years' experience as a Prompt Engineer. Proven experience tuning Large Language Models (LLMs) for strict, structured outputs, complex classification tasks, and familiarity with chain-of-thought and few-shot learning.
  • Data Analysis: Strong proficiency in identifying error patterns, analyzing model performance, and using SQL or other data analytics tools.
  • Technical Agility: Ability to quickly learn and master proprietary tools with minimal supervision.
  • Communication: Excellent verbal and written communication skills.
Optional / Preferred Skills
  • Familiarity with enterprise-grade LLM interfaces like the Goose API.
  • Experience in AI model evaluation, data science, computational linguistics, or software engineering.
  • Hands-on experience with Automated Prompt Optimization (APO) systems or tuning workflows.
  • Linguistic expertise, including an understanding of semantics and logic.

Is this role fully remote or office-based? The job requires candidates to be based in Germany, but no office location is specified. The work involves running automated tools and analyzing model outputs, which is largely compatible with remote work within Germany. What language skills are required for this position? Candidates must have native fluency in German and be fluent in English. Both languages are essential for the role. What level of experience is needed to apply? At least 4 years of experience as a Prompt Engineer is required, along with a bachelor's, master's, or doctorate degree in a relevant field such as Computer Science, Data Science, or Computational Linguistics. Is salary information available for this role? No salary or compensation details are provided in the job posting. Candidates are encouraged to discuss compensation expectations directly with the employer. What technical tools or skills should I highlight in my application? Highlight experience with LLM tuning, Automated Prompt Optimization (APO) systems, SQL-based data analysis, and familiarity with chain-of-thought and few-shot learning techniques. Experience with enterprise LLM APIs like Goose API is a plus.

Remote work & career advice that actually works.

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Project Lion - Senior Prompt Engineer - Germany (Remote, Part-Time)
Project Lion - Senior Prompt Engineer - Germany (Remote, Part-Time)

Welo Data • United States

Remote
USD 72,000 - 110,000
Project Lion - Senior Prompt Engineer - Netherland (Remote, Part-Time)
Project Lion - Senior Prompt Engineer - Netherland (Remote, Part-Time)

Remote • San Francisco (CA)

Remote
USD 120,000 - 180,000
Project Lion - Senior Prompt Engineer - Spain (Remote, Part-Time)
Project Lion - Senior Prompt Engineer - Spain (Remote, Part-Time)

Welo Data • United States

Remote
USD 77,000 - 121,000
Project Lion - Senior Prompt Engineer - France (Remote, Part-Time)
Project Lion - Senior Prompt Engineer - France (Remote, Part-Time)

Remote • San Francisco (CA)

Remote
USD 39,000 - 92,000
Project Lion - Senior Prompt Engineer - France (Remote, Part-Time)
Project Lion - Senior Prompt Engineer - France (Remote, Part-Time)

Welo Data • United States

Remote
USD 120,000 - 190,000
Project Lion - Senior Prompt Engineer - Portugal (Remote, Part-Time)
Project Lion - Senior Prompt Engineer - Portugal (Remote, Part-Time)

Remote • San Francisco (CA)

Remote
USD 67,000 - 101,000
Project Lion - Senior Prompt Engineer - Spain (Remote, Part-Time)
Project Lion - Senior Prompt Engineer - Spain (Remote, Part-Time)

Remote • San Francisco (CA)

Remote
USD 100,000 - 145,000
Project Lion - Senior Prompt Engineer - United States (Remote, Part-Time)
Project Lion - Senior Prompt Engineer - United States (Remote, Part-Time)

Remote • San Francisco (CA)

Remote
USD 140,000 - 210,000
Project Lion - Senior Prompt Engineer - Netherland (Remote, Part-Time) — Netherlands
Project Lion - Senior Prompt Engineer - Netherland (Remote, Part-Time) — Netherlands

Welo Data • United States

Remote
USD 99,000 - 143,000
Project Lion - Senior Prompt Engineer - Italy (Remote, Part-Time)
Project Lion - Senior Prompt Engineer - Italy (Remote, Part-Time)

Welo Global • Town of Italy (NY)

Remote
USD 90,000 - 130,000