Prompt Architect: AI Agent Tuning & Quality

OMEGA, Inc.; United Kingdom; Remote; GBP 45,000 - 75,000; Full time

Job summary

OMEGA, Inc. seeks an AI Prompt Engineer to shape the behavior of LLM‑driven agents in production, developing prompts, evaluations, and self‑optimizing feedback loops to keep outputs accurate, on‑brand, and cost‑efficient.

You will own prompt architectures, run evaluation suites and A/B tests, and work with the Commercial Digital team under the Head of Commercial Digital to deliver reliable, compliant results.

Qualifications

• 1+ year of hands-on experience with LLM applications, prompting, and structured outputs.
• Experience with systematic testing, data validation, or QA of AI/ML systems.
• Strong prompt-writing craft and disciplined experimentation.

Responsibilities

• Develop and tune prompts: craft and maintain prompt architectures, including system prompts, tool instructions, guardrails, and output schemas for production AI agents.
• Evaluate and test: build and execute evaluation suites and regression tests; define quality metrics and thresholds.
• Optimize performance: design A/B testing for prompt and model variants using feedback loops.
• Drive cost efficiency: support LLM optimization focusing on context window management, structured outputs, and token efficiency.
• Guard quality and tone: maintain tone and multi-language consistency for customer-facing communications in British English.
• Risk mitigation: stress-test guardrails against adversarial inputs and hallucinations.
• Document and collaborate: document patterns, test results for business users and reviewers.

Skills

LLM prompting

Evaluation pipelines

Python basics

Data handling

English writing

Tools

Azure OpenAI

Copilot Studio

Azure AI Foundry

Job description

OMEGA, Inc. seeks an AI Prompt Engineer to shape the behavior of LLM‑driven agents in production, developing prompts, evaluations, and self‑optimizing feedback loops to keep outputs accurate, on‑brand, and cost‑efficient.

You will own prompt architectures, run evaluation suites and A/B tests, and work with the Commercial Digital team under the Head of Commercial Digital to deliver reliable, compliant results.