AI Harness Engineer

Pathnovo; Bengaluru; On-site; INR 3,500,000 - 5,200,000; Full time

Job summary

Pathnovo in Bengaluru, India is seeking an AI Harness Engineer to build production-grade AI systems and the infrastructure that makes AI agents reliable and observable in real-world applications.

The role targets engineers with a strong backend and distributed systems background who have shipped AI products, built multi-agent workflows, and matured AI evaluation practices. You will work across the stack to deliver customer-facing AI capabilities.

Qualifications

• Production-grade AI product experience in real-world applications.
• Strong software engineering foundation across backend to AI evaluation.
• Experience building and operating AI systems, not just prototypes.

Responsibilities

• Build and scale production-grade AI systems and AI-powered products.
• Design robust backend services and distributed systems for AI workloads.
• Develop AI harnesses, evaluation frameworks, and testing infra.
• Create multi-agent workflows and agent orchestration.
• Collaborate with product and engineering to ship customer-facing AI features.
• Improve AI systems using experiments, metrics, and observability.

Skills

Backend engineering

Python

TypeScript

LLM evaluation

Multi-agent systems

Graph databases

Tools

LangSmith

Langfuse

Braintrust

Arize/Phoenix

Ragas

Job description

We are looking for an AI Harness Engineer who can build and scale production-grade AI systems and the infrastructure that makes AI agents reliable, observable, and effective in real-world applications. This role is ideal for someone with a strong backend and distributed systems foundation who has moved beyond experimentation into applied AI/genAI product development. You should have hands-on experience building AI products, multi-agent systems, LLM evaluation frameworks, and production AI workflows.

Responsibilities

• Build and scale production-grade AI systems and AI-powered products from 01 and beyond.
• Design robust backend services and distributed systems supporting AI/agentic workloads.
• Build AI harnesses, evaluation frameworks, and testing infrastructure to measure LLM and agent performance.
• Develop and evaluate multi-agent workflows, tool-use systems, and agent orchestration.
• Work with LLMs, GenAI, RAG, agentic AI, and applied AI systems in production.
• Design systems for LLM evaluation, observability, reliability, latency, and quality measurement.
• Build and work with graph databases for knowledge representation, relationships, agent memory, and reasoning workflows.
• Develop backend components and AI infrastructure primarily using Python and TypeScript.
• Design systems capable of handling scale, concurrency, fault tolerance, retries, and distributed execution.
• Collaborate closely with product and engineering teams to convert AI capabilities into real customer-facing products.
• Continuously improve AI systems through experimentation, evaluation, feedback loops, and production metrics.

Requirements

• We are looking for an engineer who is not just an LLM API integrator.
• The ideal candidate has a strong software engineering foundation and has actually built, shipped, and operated an AI product in production.
• You should be comfortable working across the stack, from distributed backend systems and data infrastructure to LLM evaluation, agent orchestration, and production AI reliability.

Must-Have Skills

• 3-5.5 years of software engineering experience.
• Strong backend engineering experience with distributed systems.
• Strong hands-on experience with Python and TypeScript.
• Production experience building GenAI / applied AI systems.
• Strong understanding of LLM evaluation and AI quality measurement.
• Experience building multi-agent systems / agentic AI workflows.
• Experience with graph databases and graph-based architectures.
• Experience building and operating production AI systems, not just prototypes or POCs.
• Strong understanding of APIs, microservices, databases, asynchronous processing, and distributed architectures.
• Experience building an AI-powered product end-to-end is highly preferred.

Strongly Preferred

• Experience working at a Series A or Series B startup, particularly in a high-ownership engineering environment.
• Experience taking an AI product from 01 to production.
• Experience with LLM providers, RAG, embeddings, vector databases, tool calling, and agent orchestration.
• Familiarity with AI evaluation tools/frameworks such as LangSmith, Langfuse, Braintrust, Arize/Phoenix, Ragas, DeepEval, or equivalent.
• Experience with agent memory, knowledge graphs, graph-based retrieval, or reasoning systems.
• Experience building AI systems with strong observability, tracing, evaluation, and monitoring.
• Experience working with multiple LLMs/models and optimizing quality, latency, and cost.