Research Associate - AI Evaluations

BIG City Green Residency; Bengaluru; Presencial; INR 700.000 - 1.100.000; Jornada completa

Descripción de la vacante

Karya is building AI evaluation systems and seeks a Research Associate to design and run community-facing research that shapes how AI systems are evaluated. You will contribute to end-to-end workstreams: research design, toolkits, fieldwork, and turning learnings into evaluation frameworks and datasets.

The role suits a social scientist who cares about gender equity and working with communities. You will collaborate with engineers to ensure research informs product and evaluation outcomes.

Formación

• 2-3 years of research experience in tech, impact or AI-focused org
• Degree in a social science discipline or equivalent experience
• Grounding in gender and intersectionality (caste, class, region)
• Experience in research design and delivering projects
• Hands-on qualitative and field research incl. FGDs
• Working data skills: SQL, quantitative analysis; some coding (Python or R)
• Fluency in spoken and written Hindi
• Willingness to travel for fieldwork

Responsabilidades

• Contribute to research design: questions, methods, site sampling with project leads
• Build and adapt research toolkits: guides, consent scripts, training materials
• Plan and conduct FGDs, interviews, pilots with communities; recruit participants
• Coordinate with field teams to ensure consent-first approach
• Analyze qualitative and quantitative data with gender lens
• Synthesize insights into evaluation frameworks and datasets
• Support desk research, reports and external outputs
• Assist project management by tracking timelines and deliverables

Conocimientos

Applied AI/ML literacy

Socio-technical systems thinking

Hindi & English communication

Ambiguity tolerance

Educación

Social science degree

Herramientas

SQL

Python

Descripción del empleo

Job Summary

At Karya, we re building AI evaluation systems that measure and understand whether AI systems actually work for the communities they re meant to serve. We re looking for a Research Associate to help design and run community-facing research that shapes how AI systems are evaluated. You ll contribute to research workstreams end to end: shaping the research design, building toolkits, running fieldwork, and turning what you learn into evaluation frameworks and datasets. You ll contribute to a range of evaluation projects across the team. These could include building a gender-bias benchmark for AI chatbots in domains like agriculture; conducting grounded research into what people in India actually use AI for, in which languages and in what contexts, to build evaluation categories from real use; or contributing to conceptual and methods work on what community and participatory should mean in AI evaluation, pressure-tested against fieldwork with Karya s worker community.

The role suits someone with roots in social science who has worked where technology, impact and AI meet, and who cares about gender equity and doing research with communities.

Concretely, this means:

• Contribute to research design, including research questions, methods, site selection and sampling, working closely with project leads
• Build and adapt research toolkits: discussion guides, oral consent scripts in local languages, training materials and feedback tools
• Plan and conduct focus group discussions, interviews and pilots with communities, including participant recruitment and field logistics
• Coordinate with field teams during data collection, keeping research consent-first and respectful of participants time and context
• Clean, cull and analyse qualitative and quantitative data through a gender and intersectional lens
• Synthesise insights into findings that strengthen evaluation frameworks, datasets and toolkits
• Contribute to desk research, reports and external outputs
• Support project management by tracking timelines, deliverables and decisions across internal teams (Ops, Tech) and external partners

Role Context

You ll be embedded with the Evaluations team within Karya s tech team, where social science and engineering work as one practice. We believe you can t judge whether an AI system works for people without understanding both the people and the system, and how social and technical structures shape each other. So researchers and engineers on our team work closely together, each bringing what the other can t. Your research will shape what gets built: community insights become evaluation categories, fieldwork defines what a good answer looks like, and research design decides whose voices go into a dataset. Working alongside engineers, you ll learn how AI systems are built, tested and deployed, and how they reach the communities we work with. If you re a social scientist who wants to understand AI from the inside, this is a good place to do it.

Qualifications

• At least 2-3 years of research experience, ideally at a tech, impact or AI-focused organisation
• A degree in a social science discipline (e.g., sociology, development studies, gender studies, anthropology), or equivalent experience
• Grounding in gender and intersectionality (caste, class, region) through study, research or practice
• Experience contributing to research design and delivering research projects
• Hands-on qualitative and field research experience, including running FGDs
• Working data skills: SQL, quantitative analysis, and some coding (e.g., Python or R)
• Fluency in spoken and written Hindi
• Willingness to travel for fieldwork

Skills And Competencies

• Applied AI/ML literacy: enough to engage meaningfully with model evaluation, human-in-the-loop systems and language technologies
• Socio-technical systems thinking, meaning you understand technology and society as shaping each other rather than as separate concerns
• Strong written and verbal communication in Hindi and English
• Comfort working in ambiguous, early-stage research settings

Good to have

• Proficiency in Indian languages beyond Hindi
• Familiarity with agriculture or rural livelihoods
• Exposure to participatory methods