Lead AI Harness Architect

Avensys Consulting; Singapore; On-site; SGD 150,000 - 190,000; Full time

Benefits offered by this job

Competitive compensation

Career progression

Collaborative environment

Job summary

Avensys Consulting in Singapore is seeking a Senior AI Harness Engineer to lead design, development, and evaluation of AI/LLM harnesses for testing and production use.

You will mentor engineers, collaborate with data scientists and product teams, implement CI/CD for AI testing, and advance monitoring, governance, and reliable multi-agent workflows. Experience with RAG, embeddings, vector databases, and prompt engineering is essential to deliver scalable AI solutions.

Qualifications

• Strong programming experience in Python.
• Experience building AI/LLM evaluation or testing harnesses.

Responsibilities

• Lead the design and development of AI/LLM harness frameworks for testing, evaluation, validation, and monitoring.
• Develop automated frameworks to evaluate LLM responses, AI agents, RAG pipelines, and agentic workflows.
• Design test scenarios covering accuracy, relevance, hallucination, safety, latency, reliability, and performance.
• Build evaluation pipelines using Python and modern AI/ML frameworks.
• Develop and integrate LLM-as-a-Judge, benchmark datasets, evaluation metrics, and automated scoring mechanisms.
• Create test harnesses for multi-agent systems, tool calling, function calling, and autonomous workflows.
• Work with Generative AI, RAG, embeddings, vector databases, and prompt engineering.
• Integrate AI harnesses with CI/CD pipelines to enable continuous AI testing and regression testing.
• Establish best practices for AI quality, observability, governance, and responsible AI.
• Analyze evaluation results and identify areas for improving model prompts, workflows, retrieval strategies, and agent behavior.
• Lead technical discussions and mentor engineers working on AI/ML and GenAI solutions.

Skills

Python

Generative AI

LLMs

AI Agents

AI evaluation frameworks

LangChain

LangGraph

LlamaIndex

RAG

Prompt engineering

CI/CD

Tools

Docker

Kubernetes

Git

CI/CD tooling

Cloud platforms

Job description

Avensys Consulting in Singapore is seeking a Senior AI Harness Engineer to lead design, development, and evaluation of AI/LLM harnesses for testing and production use.

You will mentor engineers, collaborate with data scientists and product teams, implement CI/CD for AI testing, and advance monitoring, governance, and reliable multi-agent workflows. Experience with RAG, embeddings, vector databases, and prompt engineering is essential to deliver scalable AI solutions.