Lead Prompting & Evaluation Engineer (NLP)

Recruitment Intelligence; Islamabad; On-site; PKR 1,800,000 - 3,200,000; Full time

Benefits offered by this job

Medical benefits

Free lunch

Annual increments

Recognition awards

Lean and agile culture

Job summary

Convo is seeking a Prompt & Evaluation Engineer with 5+ years of experience in applied NLP/LLM engineering to design prompt chains, agent behaviors, and evaluation frameworks. The role centers on translating business requirements into reliable AI solutions using Python-based evaluation, structured outputs, regression testing and thorough failure analysis, collaborating with domain experts.

You will work with cross-functional teams to optimize prompts and maintain robust evaluation pipelines

Qualifications

• 5+ years in applied NLP/LLM engineering with real-world deployments.
• Experience building evaluation or regression workflows.
• Ability to translate business use cases into AI solutions.
• Strong knowledge of prompt chains and agent behaviors.

Responsibilities

• Design modular prompt chains and structured-output contracts for commercial analysis and advisory workflows.
• Encode agent constitutions, business constraints and escalation rules defined by the Guild.
• Create representative evaluation sets, scoring rubrics and regression benchmarks.
• Implement Python-based evaluation harnesses and automated failure analysis.
• Version prompts and evaluation assets, track changes and prevent regressions across model updates.
• Analyze errors with domain experts and convert findings into prompt, data or workflow improvements.

Skills

Prompt engineering

LLM evaluation

Python

Structured outputs

Regression testing

Failure analysis

Business requirements translation

KPIs & analytics

Tools

Git

Job description

Convo is seeking a Prompt & Evaluation Engineer with 5+ years of experience in applied NLP/LLM engineering to design prompt chains, agent behaviors, and evaluation frameworks. The role centers on translating business requirements into reliable AI solutions using Python-based evaluation, structured outputs, regression testing and thorough failure analysis, collaborating with domain experts.

You will work with cross-functional teams to optimize prompts and maintain robust evaluation pipelines