AI Safety & Evaluation Lead

ConveGenius; Dadri; On-site; INR 3,500,000 - 7,000,000; Full time

Benefits offered by this job

Health insurance

Employee-friendly benefits

Growth opportunities

Job summary

ConveGenius is seeking an AI Safety & Evaluation Lead to own the framework ensuring AI systems are accurate, safe, and compliant for 150+ million learners across India. You will steer risk assessment, safety guardrails, and child-safe content practices with a focus on trust and responsible deployment.

The role demands hands-on execution, strong ownership, and the ability to navigate ambiguity while building scalable evaluation systems and governance artifacts in a growing education tech context.

Qualifications

• Experience designing and executing model evaluation frameworks and test methodologies.
• Proven ability to perform red-teaming and adversarial testing on AI systems.
• Strong understanding of bias, fairness, hallucination, and content safety in AI.
• Ability to measure model accuracy, grounding, and response quality.
• Knowledge of AI governance, compliance, and responsible AI practices.

Responsibilities

• Design and own the AI evaluation framework: benchmark construction and eval pipelines.
• Lead red-teaming and adversarial testing across deployed models.
• Define AI safety standards and guardrails for scalable deployment.
• Build evaluation tooling and integrate governance checkpoints in release pipelines.
• Conduct systematic bias detection and fairness assessments across outputs.
• Translate safety findings into actionable model or prompt changes.
• Develop model cards, audit trails, and safety reports for stakeholders.
• Collaborate with engineering on governance integration in deployments.
• Keep updated with EU/INDIA/NIST policy frameworks for compliance.
• Measure grounding fidelity and response quality across systems.
• Document AI governance practices and enable scalable operations.
• Stay hands-on during high-velocity setup phases and team-building.

Skills

AI safety

Evaluation framework

Red-teaming

Bias & fairness

Model accuracy

Governance & compliance

NIST RMF

Student psychology

Production AI

Problem solving

Tools

RAGAS

LangSmith

PromptBench

EleutherAI lm-evaluation-harness

Job description

Does working for 150+ million children of Bharat excite you?

Then this opportunity is for you!

About us:

ConveGenius is an AI-native education infrastructure company transforming how governments, enterprises, and social institutions deliver learning and governance outcomes at population scale. Operating across B2G and B2B segments, the company partners with state governments, CSR funders, philanthropies, and social enterprises to build intelligent education ecosystems that are both impactful and commercially sustainable.

At its core, ConveGenius runs two synergistic business lines:

• SwiftAI – the intelligence and automation layer that powers citizen engagement, governance efficiency, and real‑time decision‑making for governments and institutions.
• SwiftSchool – the learning infrastructure layer that integrates adaptive learning, digital classrooms, and analytics to drive measurable student and system outcomes.

This dual‑engine model enables ConveGenius to serve as the technology backbone for education reform — connecting classrooms, communities, and command centres through a unified AI‑powered framework. Unlike traditional edtech companies that deliver content, ConveGenius builds infrastructure — platforms that integrate deeply into public delivery systems while enabling private and philanthropic partners to scale their missions with measurable results.

With operations across 22+ Indian states and partnerships with multilaterals, CSR alliances, and government departments, ConveGenius has established itself as a partner‑friendly enterprise trusted for aligning policy priorities with social and financial returns. Supported by institutional investors and guided by a double bottom line philosophy, the company consistently delivers impact at scale and profitability with purpose.

As ConveGenius enters its next phase of growth, it is shaping the future of AI‑led public education infrastructure — where technology, partnerships, and governance intelligence converge to redefine how nations learn and thrive.

Role Overview: As the AI Safety & Evaluation Lead at ConveGenius, you will own the process ensuring every AI system deployed is accurate, safe, unbiased, secure, and compliant. Given that ConveGenius serves children and underserved communities across India, this role carries direct responsibility for learner trust, content safety, and regulatory standing. A core dimension of this role is understanding student psychology, since our entire user base is school and college‑going students, and AI outputs must be designed to handle sensitive or negative queries (e.g. distressing searches by students) in a safe, age‑appropriate manner that does not surface harmful information. This role is part of a 0‑to‑1 vertical build. The person joining should be comfortable with ambiguity, high ownership, rapid iteration, hands‑on execution, team building, and periods of high‑intensity work during the early setup phase.

Key Responsibilities

• Design and own the AI evaluation framework: benchmark construction, automated and human‑in‑the‑loop eval pipelines.
• Lead red‑teaming and adversarial testing across deployed models: prompt injection, jailbreaking, edge case identification.
• Define AI safety standards, responsible AI policies, and child‑safe content guardrails for deployment at scale.
• Build and maintain evaluation tooling: RAGAS, LangSmith, PromptBench, EleutherAI lm‑evaluation‑harness.
• Conduct systematic bias detection and fairness measurement across outputs, especially for regional and linguistic variation.
• Manage AI risk assessments and translate safety findings into actionable model or prompt changes.
• Establish AI governance documentation: model cards, audit trails, safety reports for internal and regulatory stakeholders.
• Embed evaluation and safety checkpoints into the model release pipeline in collaboration with engineering.
• Track and apply evolving regulatory requirements: EU AI Act, NIST AI RMF, India AI policy frameworks.
• Measure and report model accuracy, grounding fidelity, and response quality across deployed AI systems through systematic evaluation.
• Build and maintain knowledge of AI governance and compliance standards — operationalising responsible AI practices into daily engineering workflows.

Must‑Have Skills

• Strong experience in AI/LLM safety, evaluation, and risk assessment
• Expertise in designing and executing model evaluation frameworks and test methodologies
• Experience in red‑teaming, adversarial testing, and identifying model vulnerabilities
• Strong understanding of bias, fairness, hallucination, and content safety in AI systems
• Experience measuring model accuracy, grounding, and response quality
• Knowledge of AI governance, compliance, and responsible AI practices
• Familiarity with frameworks such as NIST AI RMF and AI regulatory standards; understanding of student psychology and age‑appropriate AI output design
• Strong analytical and problem‑solving skills with experience working on production AI systems

What We Offer and Why You Should Join ConveGenius

At ConveGenius, we create an environment where purpose meets growth. You will be part of a mission‑driven organization that is transforming how learning reaches children across India, especially at scale and where it matters most.

• Work with a diverse and passionate team spread across India, bringing together people from different backgrounds, perspectives, and expertise.
• Be part of an organization with a strong on ground presence across more than two thirds of India, giving you exposure to large scale, real world impact.
• Play a meaningful role in strengthening the education ecosystem and contributing directly to learning outcomes for millions of children.
• Gain experience working at the intersection of education, technology, and public systems, building skills that are highly relevant and future focused.
• Work in a culture that values ownership, learning, and collaboration, with space to take initiative and grow professionally.
• Enjoy stability and peace of mind with employee friendly benefits, including health insurance coverage and other supportive policies.
• If you are driven by purpose, enjoy solving complex problems, and want your work to create visible impact, ConveGenius offers a place to grow while making a difference.