Technical Program Manager AI Benchmarking

Precision Development (PxD); New Delhi; On-site; INR 4,000,000 - 6,000,000; Full time

Job summary

Precision Development (PxD) seeks a Technical Program Manager to lead a national AI benchmarking initiative for agricultural advisory in India. You will coordinate government agencies, academics, and partners to design and run a rigorous evaluation framework and deliver decision-ready evidence.

Based in New Delhi with travel to partner institutions and project states, you will manage governance, data access, and vendor relationships while turning results into policy-relevant recommendations for

Qualifications

• Masters degree (or equivalent experience) in Computer Science, Data Science, Engineering, Public Policy, Economics, Agriculture, Development Studies, Management, or another relevant field.
• 5–8 years of progressively responsible experience managing complex technical programs or projects.
• Working fluency with modern AI systems — LLMs, retrieval-augmented generation, prompting, model evaluation, and the practical trade-offs between accuracy, latency, safety, and cost.
• Demonstrated experience managing multi-stakeholder projects involving government agencies, academic institutions, and external technical partners.
• Strong project management skills with the ability to run parallel workstreams against fixed donor milestones.
• Ability to translate technical evidence into clear, decision-oriented recommendations for non-technical senior audiences.

Responsibilities

• Serve as PxD's day-to-day point of contact for the benchmarking program, aligning the Ministry of Agriculture and Farmers Welfare on priority use cases and models.
• Establish the governance framework for the consortium and keep it functioning through all stages.
• Negotiate system and data access, including government advisory environments, and maintain a permissions tracker across partners.
• Facilitate workshops with government counterparts and model providers; manage the funder relationship and reporting.
• Own the benchmarking framework end to end: use cases, tasks, datasets, rubrics, safety criteria, and cost templates.
• Coordinate technical benchmarking across evaluation layers and assess speech, translation, and multilingual performance.
• Oversee latency, reliability, and total cost of ownership; manage cloud compute, GPU, and API resources.
• Maintain independence and neutrality of benchmarking with transparent disclosures of conflicts of interest.
• Oversee pre-deployment user evidence generation with agronomists and farmer testers; manage IRB submissions.
• Consolidate findings into a single decision framework and translate into ministry-ready recommendations.

Skills

Stakeholder management

Program management

Policy coordination

AI/ML systems knowledge

Communication skills

Education

Master's degree in Computer Science/DS/Engineering/Public Policy

Tools

Model evaluation

LLMs

ASR/TTS

Job description

Job Description

Title: Technical Program Manager, AI Benchmarking

Location: New Delhi, India

Work Authorization: Current and ongoing work authorization in India

Position: Full-time

Application deadline: September 18, 2026, 6 pm IST (applications may be reviewed as they are submitted)

Role Overview

PxD is looking for a Technical Program Manager to lead delivery of a national initiative benchmarking Indian AI models for agricultural advisory. Government advisory platforms are increasingly powered by large language models, and a growing set of Indian foundational models, fine-tuned variants, and voice-based systems have emerged as potential alternatives — but little rigorous evidence exists on how they perform inside live advisory workflows. You will run a multi-partner benchmarking program that produces decision-ready evidence for the Ministry of Agriculture and Farmers’ Welfare on which models can be safely and effectively integrated into existing advisory systems.

PxD will co-design the benchmark framework and act as consortium facilitator and government coordination partner for this work, convening government agencies, academic technical evaluators, benchmark partners, and model providers. You will be the person who holds that consortium together day to day: setting the benchmarking architecture with government, contracting and managing academic and technical partners, overseeing the evidence pipeline from technical testing through farmer-facing user research, and turning the results into recommendations a ministry can act on. This position is based in New Delhi, with travel to partner institutions and project states.

About PxD

PxD is a rapidly scaling global non‑profit that harnesses the power of technology, data science, and behavioral economics to provide targeted information to farmers in low- and middle‑income countries. Providing the right information to the right people, in the right way and at the right time, increases incomes, protects the environment, and improves well‑being. We are evidence-based: we test our interventions with randomized controlled trials to prove our impact and cost‑effectiveness, to simultaneously learn from and improve our service, and to demonstrate our value to funders and partners.

Responsibilities

Government & Stakeholder Management

• Serve as PxDs day-to-day point of contact for the benchmarking program, working through PxDs embedded PMU team to align the Ministry of Agriculture and Farmers Welfare on priority use cases, target languages and geographies, candidate models, benchmarking baselines, and approval pathways.
• Establish the governance framework for the consortium — partner roles, decision rights, steering cadence, and escalation routes — and keep it functioning as the program moves through its stages.
• Negotiate system and data access, including access to government advisory environments for substitution testing, and maintain a permissions tracker across government, academic, and advisory data sources.
• Facilitate workshops and consultations with government counterparts, model providers, and domain experts; manage the funder relationship and reporting.

Benchmarking Design & Technical Delivery

• Own the benchmarking framework end to end: advisory use cases, benchmark tasks, datasets, human-rating rubrics, safety and language criteria, tool-use and source-grounding assessments, cost comparison templates, and decision thresholds.
• Work with existing benchmark initiatives to adapt and localise established assets rather than building a parallel benchmark.
• Coordinate technical benchmarking across three layers — standalone model evaluation, substitution testing within the live government advisory stack, and a neutral benchmarking harness where direct system access is unavailable — alongside assessment of speech recognition, translation, text‑to‑speech, and multilingual performance.
• Manage latency, reliability, and total‑cost‑of‑ownership analysis, and oversee the cloud compute, GPU, and API resources that represent the single largest cost line in the program.
• Safeguard the independence and neutrality of the benchmarking by maintaining clear separation between benchmark development, model evaluation, and model development roles of partner organisations, with transparent disclosure of potential conflicts of interest.

Evidence Generation & Synthesis

• Oversee pre‑deployment user evidence generation with agronomists, extension personnel, and compensated farmer participant testers, assessing whether advisory outputs are comprehensible, actionable, trustworthy, and usable in local languages.
• Manage IRB submission and third‑party field vendor selection and oversight for user testing.
• Consolidate findings across technical performance, user evidence, safety, operational readiness, and cost into a single decision framework, and translate them into recommendations a non‑technical government audience can act on.
• Author the final evidence synthesis and decision report identifying models ready for pilot deployment, models requiring further optimization, and key implementation risks.

Grants, Contracts & Partner Management

• Contract and manage academic and technical partners, sandbox and compute hosts, domain‑expert consultants, and field vendors, monitoring performance against agreed deliverables.
• Ensure contractual and donor compliance throughout implementation, including data‑processing agreements and dataset licenses.
• Coordinate with finance and operations teams on budgets, procurement, and financial reporting against grant milestones.

Team & Knowledge Management

• Work alongside research and field staff supporting the program, providing technical direction and day‑to‑day coordination.
• Document benchmarking tools, protocols, and methodology so future evaluations across agricultural and other public‑sector advisory systems can build on this work.
• Contribute to PxDs institutional knowledge management and cross‑country learning on AI advisory.

What Youll Bring

Required

• Masters degree (or equivalent experience) in Computer Science, Data Science, Engineering, Public Policy, Economics, Agriculture, Development Studies, Management, or another relevant field.
• 5–8 years of progressively responsible experience managing complex technical programs or projects.
• Working fluency with modern AI systems — LLMs, retrieval‑augmented generation, prompting, model evaluation, and the practical trade‑offs between accuracy, latency, safety, and cost.
• Demonstrated experience managing multi‑stakeholder projects involving government agencies, academic institutions, and external technical partners.
• Strong project management skills with the ability to run parallel workstreams against fixed donor milestones.
• Ability to translate technical evidence into clear, decision‑oriented recommendations for non‑technical senior audiences.
• Strong stakeholder management and relationship‑building abilities.
• Excellent organizational skills and exceptional attention to detail.

Preferred

• Demonstrated experience with AI model evaluation (academic/ personal/ work projects).
• Direct experience designing or running model evaluations, benchmarks, or structured A/B testing of AI systems.
• Experience with Indian-language NLP, speech technology (ASR/TTS), or multilingual model performance.
• Experience working with state or national government programs, or with digital public infrastructure.
• Background in agriculture, climate, technology, or international development.
• Familiarity with causal inference and experimental design.
• Comfort operating with significant ambiguity and limited existing infrastructure, having built or launched a new program from scratch.
• Proficiency in one or more Indian languages.

PxD is committed to building an inclusive and diverse organisation. We don’t make decisions based on gender, gender identity, class, ethnicity, sexual orientation, or any other personal characteristics that are unrelated to the role. We understand that everyone has different lifestyles and we accommodate flexible working to make sure you are as productive and happy as you can be.

We will use any personal data you provide for recruitment purposes only.