ML Researcher – Benchmarks & Evaluation

Deccan AI Automation Private Limited; Mountain View (CA); Hybrid; USD 150,000 - 200,000; Full time

Job summary

Deccan AI is seeking an ML Researcher to drive end-to-end AI research, identify emerging directions, and design novel benchmarks and evaluation datasets aligned with organizational priorities. This role bridges theory and practice by translating recent papers into actionable benchmarks for agent frameworks, deep learning, NLP, and other AI paradigms, with strong documentation and clear success criteria.

Collaborate with ML leads and project managers to deliver high-impact research outputs,

Qualifications

• Deep expertise in AI/ML domains including agent frameworks, deep learning, NLP, computer vision, and emerging AI systems.
• Proven experience designing benchmarks and evaluation datasets from the ground up.
• Excellent documentation and communication abilities.

Responsibilities

• Research & literature review: track the latest AI research, conferences, and publications.
• Design novel AI benchmarks and evaluation datasets aligned with organizational priorities.
• Provide clear problem statements, design overviews, evaluation metrics, and testing guidelines.
• Collaborate with ML leads and project managers to align outputs with goals.
• Produce high-impact research outputs with actionable insights.

Skills

AI/ML expertise

Agent frameworks

Deep learning

NLP

Computer vision

Job description

Deccan AI is a GenAI data company that runs the post-training and production layer for the world's leading AI labs and enterprises — from expert feedback and RL environments to managed AI workforces and evaluation infrastructure. Backed by A91 Partners, SIG (Susquehanna International Group), and Prosus Ventures, and having grown 10x last year, Deccan works with a majority of the Magnificent 7 and the top frontier labs globally. Its global network of 1 million+ domain experts powers Helix, its hybrid eval suite, and EnterpriseOS, its scaled ops automation platform. Headquartered in Mountain View, CA.

Job Description:

1. Background

The ML Researcher plays a critical role in keeping the organization at the forefront of AI innovation through deep, end-to-end research. This role is focused on identifying emerging research directions, designing novel AI benchmarks, and creating high-quality evaluation datasets aligned with both cutting-edge research and organizational priorities.

The ML Researcher bridges the gap between theoretical research and practical application by translating insights from the latest AI papers into actionable benchmarks and evaluation frameworks. The role emphasizes deep research, synthesis of academic and industry findings, and the design of solutions that meaningfully advance AI systems—particularly in agent frameworks, machine learning models, and emerging AI paradigms.

2. Purpose of the Role

The primary objective of the ML Researcher is to conduct comprehensive AI research and design novel, scalable, and real-world–relevant benchmarks and evaluation datasets. The role aims to push beyond existing evaluation practices by defining new criteria that better measure the capabilities and limitations of modern AI systems.

3. Key Responsibilities

1. Research & Literature Review

Continuously track the latest AI research, conferences, and publications.

Conduct in-depth literature reviews across emerging AI models, agent frameworks, and evaluation methodologies.

Identify gaps, limitations, and opportunities in existing benchmarks.

Assess current industry benchmarks for relevance to organizational and client goals.

Propose novel AI benchmarks informed by research insights and industry gaps.

Design evaluation datasets that assess AI systems across both coding and non-coding domains.

Ensure benchmarks and datasets are innovative, scalable, and practically applicable.

Define meaningful evaluation metrics that provide actionable insights into model performance.

3. Documentation & Research Deliverables

Create high-level requirement documents for each benchmark or dataset, covering:

Problem statement and motivation

Design overview and structure

Evaluation metrics and success criteria

Testing and validation guidelines

Ensure documentation is clear, comprehensive, and implementation-ready.

4. Cross-Functional Collaboration

Work closely with the ML Lead, pipeline-focused MLEs, and project managers to align research outputs with organizational priorities.

Collaborate with internal stakeholders to ensure benchmarks meet project goals and industry standards.

Provide feedback and iterative improvements based on implementation outcomes.

Refine and evolve benchmarks and datasets based on feedback, new research, and emerging use cases.

Propose enhancements to existing evaluation methods, including new metrics or benchmark variations.

Stay actively engaged with the AI research community through conferences, discussions, and ongoing learning.

The ML Researcher is expected to produce high-impact, actionable research outputs, including:

Each proposal should include:

Clear problem definition and motivation

Detailed benchmark design and structure

Defined evaluation metrics

Testing and validation guidelines

Explanation of novelty compared to existing benchmarks

2. Evaluation Dataset Designs

Dataset overview, structure, and intended use

(Optional) Data collection, labeling, and cleaning methodology

Evaluation methodology and expected outcomes

Unique or differentiating features of the dataset

3. Research Reports & Whitepapers (Optional)

Periodic summaries of research findings or emerging trends

Internal documentation of best practices and benchmarking insights

4. Feedback & Iteration Reports

Post-implementation assessments of benchmark effectiveness

Recommendations for improvements based on team and client feedback

Iterative updates aligned with new research and usage insights

5. Timeline & Milestones

Complete initial research and first benchmark + dataset proposal

Deliver at least one new benchmark and one evaluation dataset per month

Submit finalized documentation for internal review

Incorporate feedback and finalize benchmark and dataset proposals

Complete testing and validation guidelines

Ongoing:

Regularly iterate on existing benchmarks

Provide monthly research summaries on emerging AI trends impacting evaluation

6. Expected Output & Impact

Innovative Benchmarks & Datasets: Continuous delivery of novel, high-quality evaluation frameworks.

Industry Impact: Contributions that elevate evaluation standards and strengthen the organization’s research credibility.

Client Enablement: Research-driven inputs that support effective, client-facing AI solutions.

7. Seniority & Skillset Requirements

The ideal ML Researcher will have:

Deep expertise in AI/ML domains including agent frameworks, deep learning, NLP, computer vision, and emerging AI systems.

Proven experience designing benchmarks and evaluation datasets from the ground up.

Strong research and analytical skills with the ability to translate papers into practical solutions.

Excellent documentation and communication abilities.

Ability to work independently while collaborating effectively with cross‑functional teams.