SwarmBench Task Engineer (Knowledge/Research) - 75064

Turing; Pune District; On-site; INR 6,704,980 - 9,578,544; Part time

Benefits offered by this job

Fully remote environment

Cutting-edge AI projects

Performance-based contract extension

Job summary

Turing is seeking a highly analytical individual in Pune District, India, to join as a researcher. The role involves crafting challenging problems, analyzing large documents, and designing evaluation benchmarks. Candidates should have a strong research background, proficiency in Python, and experience with JSON. The position offers a fully remote environment and the opportunity to work on cutting-edge AI projects. The engagement is a contractor assignment for 1 month, requiring 40 hours/week with specific PST overlap.

Qualifications

• 5+ years of research experience in any scientific domain.
• Ability to extract structured data from unstructured text.
• Experience with JSON and schema design.
• Proficiency in Python for data processing.
• Familiarity with AI coding benchmarks like SWE-bench.
• Experience with writing Dockerfiles and debugging.

Responsibilities

• Build benchmark tasks that require analyzing large document collections.
• Curate research corpora and design comprehensive questions.
• Write structured ground-truth oracles that can be verified.
• Design prompts to evaluate agent output against oracles.
• Create guides for distributing research tasks across agents.

Skills

Research experience

Reading comprehension

Python scripting

Docker

Attention to detail

Tools

JSON

Docker

Job description

Based in San Francisco, California, Turing is the world’s leading research accelerator for frontier AI labs and a trusted partner for global enterprises deploying advanced AI systems. Turing supports customers in two ways: first, by accelerating frontier research with high-quality data, advanced training pipelines, plus top AI researchers who specialize in coding, reasoning, STEM, multilinguality, multimodality, and agents; and second, by applying that expertise to help enterprises transform AI from proof of concept into proprietary intelligence with systems that perform reliably, deliver measurable impact, and drive lasting results on the P&L.

Role Overview

We are seeking a highly analytical and computationally proficient individual to join our team with a strong research background. You will be instrumental in contributing to this role by either crafting challenging and insightful problems in your respective research domain, devising elegant computational solutions.

Responsibilities

• Build multi-agent benchmark tasks that require reading, analyzing, and synthesizing large document collections
• Curate real-world research corpora — academic papers, case studies, technical reports — and design questions that require comprehensive analysis
• Write structured ground-truth oracles (JSON) with specific, verifiable answers that prove the agent actually read the source material
• Design LLM judge prompts that evaluate agent output field-by-field against the oracle
• Create decomposition guides that split research across multiple parallel sub-agents (one per document, one per domain, then synthesis)

Required Qualifications

• 5+ years of research experience (academic or industry) in any scientific domain
• Strong reading comprehension with ability to extract structured data from unstructured text
• Experience with JSON and data structures, including schema design and output validation
• Proficiency in Python scripting for data processing and evaluation (e.g., judge scripts)
• Familiarity with AI coding benchmarks such as SWE-bench and Terminal-bench
• Hands-on experience with Docker (writing Dockerfiles, building images, debugging containers)
• High attention to detail, especially for creating precise evaluation oracles without approximations

Nice to have

• Experience with systematic reviews, meta-analyses, or large-scale literature surveys
• Familiarity with medical, legal, or scientific document analysis
• Experience with NLP or information extraction tasks
• Knowledge of LLM evaluation and benchmarking (e.g., MMLU, GPQA, SimpleQA)
• Experience curating datasets for AI evaluation

Perks of Freelancing With Turing

• Work in a fully remote environment.
• Opportunity to work on cutting-edge AI projects with leading LLM companies.
• Potential for contract extension based on performance and project needs.
• Commitments Required : 40 hours /week with 4 hours of PST Overlap
• Engagement type : Contractor assignment/freelancer (no medical/paid leave)
• Duration of contract : 1 month; [expected start date is next week]