Prompt & Evaluation Engineer — Craft AI Prompts, Metrics & Tests

Convo; Islamabad; On-site; PKR 1,800,000 - 2,400,000; Full time

Job summary

Convo is seeking a Prompt & Evaluation Engineer with 5+ years of experience in applied NLP/LLM engineering to design and optimize prompt chains, agent behaviors, and evaluation frameworks. The role emphasizes translating business requirements into reliable AI solutions through Python-driven evaluation and robust structured-output contracts.

The candidate will develop modular prompt architectures, encode business constraints, and build comprehensive evaluation suites with regression benchmarks,

Qualifications

• 3+ years in applied NLP or LLM engineering.
• Experience building systematic evaluation or regression workflows.
• Hands-on prompt engineering for multi-step, tool-using or structured-output LLM applications.

Responsibilities

• Design modular prompt chains and structured-output contracts for commercial analysis and advisory workflows.
• Encode agent constitutions, business constraints and escalation rules defined by the Guild.
• Create representative evaluation sets, scoring rubrics and regression benchmarks.
• Implement Python-based evaluation harnesses and automated failure analysis.
• Version prompts and evaluation assets, track changes and prevent regressions across model updates.
• Analyze errors with domain experts and convert findings into prompt, data or workflow improvements.

Skills

Prompt engineering

LLM evaluation

Python scripting

Structured outputs

Regression testing

Version control

Business KPI reasoning

Tools

Git

Job description

Convo is seeking a Prompt & Evaluation Engineer with 5+ years of experience in applied NLP/LLM engineering to design and optimize prompt chains, agent behaviors, and evaluation frameworks. The role emphasizes translating business requirements into reliable AI solutions through Python-driven evaluation and robust structured-output contracts.

The candidate will develop modular prompt architectures, encode business constraints, and build comprehensive evaluation suites with regression benchmarks,