Senior AI Harness Engineer
AVENSYS CONSULTING PTE. LTD.
A complete application in a minute — tailored resume and cover letter, ready to send.
Job summary
Avensys Consulting Pte. Ltd. is seeking a Senior AI Harness Engineer to design, build, and maintain frameworks, evaluation systems, testing infrastructure, and tooling for production-ready AI/LLM applications.
The role focuses on creating AI harnesses that enable systematic testing, benchmarking, observability, and continuous improvement of Generative AI and Agentic AI systems. Strong Python and AI evaluation expertise are required for enterprise-scale projects.
Qualifications
- 8+ years of software/engineering experience.
- Experience in AI/ML or Generative AI engineering is required.
- Strong hands-on experience with Python and AI/LLM technologies.
- Proven experience developing AI evaluation, testing, automation, or LLMOps solutions.
- Experience with enterprise-scale AI applications and production deployments.
Responsibilities
- Design and implement AI/LLM evaluation and testing harnesses for Generative AI and Agentic AI applications.
- Build reusable frameworks for model evaluation, prompts, regression testing, benchmarking, and performance validation.
- Develop automated test suites to assess accuracy, relevance, groundedness, toxicity, and latency of AI systems.
- Create testing frameworks for multi-agent workflows, RAG pipelines, and tool-calling apps.
- Develop Python-based AI engineering frameworks, utilities, SDKs, and automation tools.
- Implement MLOps/LLMOps pipelines for model, prompt, dataset, and evaluation lifecycle management.
Skills
Education
Tools
Job description
Avensys is a reputed global IT professional services company headquartered in Singapore. Our service spectrum includes enterprise solution consulting, business intelligence, business process automation and managed services. Given our decade of success we have evolved to become one of the top trusted providers in Singapore and service a client base across banking and financial services, insurance, information technology, healthcare, retail, and supply chain.
We are currently looking to hire Senior AI Harness Engineer.
This is an exciting opportunity to expand your skill set, achieve job satisfaction and work-life balance. More details as below.
We are looking for a Senior AI Harness Engineer to design, build, and maintain the engineering frameworks, evaluation systems, testing infrastructure, and tooling required to develop reliable, scalable, and production-ready AI/LLM applications. The role will focus on building AI harnesses that enable systematic testing, evaluation, benchmarking, observability, and continuous improvement of Generative AI and Agentic AI systems.
The ideal candidate will have strong experience in Python, LLMs, Generative AI, AI agents, evaluation frameworks, RAG, prompt engineering, API integration, cloud platforms, and MLOps/LLMOps, with the ability to build robust engineering solutions around AI models.
- Design and develop AI/LLM evaluation and testing harnesses for Generative AI and Agentic AI applications.
- Build reusable frameworks for model evaluation, prompt testing, regression testing, benchmarking, and performance validation.
- Develop automated test suites to evaluate accuracy, relevance, groundedness, hallucination, toxicity, safety, latency, cost, and response quality.
- Create testing frameworks for LLM-based agents, multi-agent workflows, RAG pipelines, and tool-calling applications.
- Develop and maintain Python-based AI engineering frameworks, utilities, SDKs, and automation tools.
- Implement LLMOps/MLOps pipelines for model, prompt, dataset, and evaluation lifecycle management.
- Build automated CI/CD pipelines for AI applications, including model and prompt regression testing.
- Integrate AI evaluation tools and frameworks such as MLflow, LangSmith, DeepEval, Ragas, Azure AI evaluation, OpenAI evaluation frameworks, or equivalent technologies.
- Design evaluation datasets, test cases, golden datasets, benchmark datasets, and synthetic test data.
- Develop automated mechanisms for prompt/version management and experiment tracking.
- Implement observability and monitoring for AI applications, including LLM traces, token usage, latency, errors, model performance, and cost.
- Develop harnesses for testing RAG systems, including retrieval quality, chunking strategies, embeddings, vector search, reranking, and grounded responses.
- Build evaluation capabilities for function calling, API/tool invocation, MCP-based integrations, and agentic workflows.
- Work with AI/ML engineers and application teams to identify failure patterns and improve model/application performance.
- Establish engineering standards for AI quality, reliability, security, scalability, and responsible AI.
- Troubleshoot complex issues across application code, LLM APIs, vector databases, cloud services, and AI infrastructure.
- Build dashboards and reports to communicate AI evaluation and quality metrics to technical stakeholders.
- Mentor junior engineers and contribute to architecture and technical design decisions.
- Strong hands-on experience with Python.
- Good knowledge of REST APIs, JSON, asynchronous programming, SDK development, and microservices.
- Experience with Git, GitHub/GitLab/Bitbucket, code reviews, and software engineering best practices.
- Strong understanding of LLMs and Generative AI application architecture.
- Experience with models such as OpenAI, Azure OpenAI, Anthropic, Gemini, Llama, Mistral, or equivalent.
- o Prompt Engineering
- o RAG
- o Embeddings
- o Vector databases
- o Function/Tool Calling
- o Structured outputs
- o Agentic AI
- o LLM APIs
- o Context management
- o Model evaluation
- LangChain
- LangGraph
- LlamaIndex
- Semantic Kernel
- AutoGen
- CrewAI
- OpenAI Agents SDK
- MCP (Model Context Protocol)
- Ragas
- DeepEval
- LangSmith
- MLflow
- Azure AI Evaluation
- OpenAI evaluation frameworks
- Custom LLM evaluation frameworks
- Accuracy
- Precision / Recall
- Relevance
- Faithfulness
- Groundedness
- Context relevance
- Hallucination detection
- Toxicity and safety
- Robustness
- Latency
- Token consumption
- Cost per request
- Experience designing and testing RAG pipelines.
- Knowledge of vector databases such as:
- o Azure AI Search
- o Pinecone
- o Weaviate
- o Milvus
- o Qdrant
- o FAISS
- o Elasticsearch/OpenSearch
- Experience evaluating retrieval quality, embeddings, reranking, chunking, and search strategies.
- Microsoft Azure
- AWS
- Google Cloud Platform
- Docker
- Kubernetes
- CI/CD
- GitHub Actions / Azure DevOps / Jenkins
- Infrastructure automation
- Cloud monitoring and logging
- Secrets and configuration management
- Experience implementing MLOps/LLMOps practices.
- Model and prompt versioning.
- Experiment tracking.
- Dataset management.
- Automated evaluation pipelines.
- Continuous monitoring.
- Model/application performance tracking.
- Production deployment and rollback strategies.
- Experience building AI testing platforms or internal AI developer tools.
- Experience with agent evaluation and multi-agent systems.
- Experience with MCP servers and MCP-based tool integrations.
- Knowledge of AI security, prompt injection, jailbreak testing, data leakage, and adversarial evaluation.
- Experience with synthetic data generation and automated test-case generation.
- Knowledge of responsible AI and AI governance.
- Experience with SQL and NoSQL databases.
- Familiarity with Kafka or event-driven architectures.
- Experience with TypeScript/Node.js is an advantage.
- Experience working in Agile/Scrum environments.
The Senior AI Harness Engineer will be expected to deliver:
- Reusable AI testing and evaluation frameworks.
- Automated LLM and agent regression test suites.
- AI quality and performance dashboards.
- Automated evaluation pipelines integrated with CI/CD.
- Benchmark and golden datasets.
- RAG and agent evaluation capabilities.
- Prompt/model comparison frameworks.
- AI observability and monitoring solutions.
- Production-ready engineering standards for GenAI applications.
- 8+ years of overall software/engineering experience.
- 4+ years of experience in AI/ML or Generative AI engineering.
- Strong hands-on experience with Python and AI/LLM technologies.
- Proven experience developing AI evaluation, testing, automation, or LLMOps solutions.
- Experience working with enterprise-scale AI applications and production deployments.
- Strong analytical and problem-solving skills.
- Ability to translate AI quality requirements into measurable engineering solutions.
- Strong communication and collaboration skills.
- Ability to work with AI researchers, ML engineers, software developers, DevOps engineers, and business stakeholders.
- Strong ownership mindset with a focus on building reliable and maintainable AI systems.
- Ability to mentor engineers and provide technical leadership.
Bachelor's or Master's degree in Computer Science, Artificial Intelligence, Data Science, Engineering, or a related field.
'/%3e%3cpath%20d='M101.744%2012.4726L102.894%2016.1947L99.0021%2013.2201L101.744%2012.4726ZM105.3%208.40937H100.488L99.0026%203.59473L97.5116%208.41047L92.7%208.40385L96.5965%2011.3834L95.1055%2016.1942L99.0021%2013.2196L101.408%2011.3834L105.3%208.40992V8.40937Z'%20fill='white'/%3e%3cdefs%3e%3clinearGradient%20id='paint0_linear_5593_1381'%20x1='90'%20y1='9.44471'%20x2='99'%20y2='9.44471'%20gradientUnits='userSpaceOnUse'%3e%3cstop%20stop-color='%2300AD70'/%3e%3cstop%20offset='0.998594'%20stop-color='%2300B67A'/%3e%3cstop%20offset='1'%20stop-color='%23DCDCE6'/%3e%3c/linearGradient%3e%3c/defs%3e%3c/svg%3e){kind=small}
