Senior AI Harness Architect for LLM Testing & MLOps

AVENSYS CONSULTING PTE. LTD.

Singapore

On-site

SGD 150,000 - 190,000

Full time

14 days+

Application generator

An application made for this job — a tailored resume and cover letter that speak straight to the posting.

Get past ATS filters

Job summary

Avensys Consulting Pte. Ltd. is seeking a Senior AI Harness Engineer to design, build, and maintain frameworks, evaluation systems, testing infrastructure, and tooling for production-ready AI/LLM applications.

The role focuses on creating AI harnesses that enable systematic testing, benchmarking, observability, and continuous improvement of Generative AI and Agentic AI systems. Strong Python and AI evaluation expertise are required for enterprise-scale projects.

Qualifications

8+ years of software/engineering experience.
Experience in AI/ML or Generative AI engineering is required.
Strong hands-on experience with Python and AI/LLM technologies.
Proven experience developing AI evaluation, testing, automation, or LLMOps solutions.
Experience with enterprise-scale AI applications and production deployments.

Responsibilities

Design and implement AI/LLM evaluation and testing harnesses for Generative AI and Agentic AI applications.
Build reusable frameworks for model evaluation, prompts, regression testing, benchmarking, and performance validation.
Develop automated test suites to assess accuracy, relevance, groundedness, toxicity, and latency of AI systems.
Create testing frameworks for multi-agent workflows, RAG pipelines, and tool-calling apps.
Develop Python-based AI engineering frameworks, utilities, SDKs, and automation tools.
Implement MLOps/LLMOps pipelines for model, prompt, dataset, and evaluation lifecycle management.