Senior Software Engineer — AI Evaluation & Benchmarks

Alignerr Corp.

India

Remote

INR 1,653,000 - 4,133,000

Part time

23 hours ago
Be an early applicant

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Benefits offered by this job

Fully remote
3-month contract
Potential extension

Job summary

Alignerr is seeking experienced Software Engineers to design and build coding benchmarks and data pipelines for evaluating frontier AI models. You will work across real-world programming tasks, assess AI-generated code, and help shape how state-of-the-art systems reason and produce production-quality software.

This fully remote hourly contract role spans 3 months, with potential for extension. If you thrive in fast-paced AI research environments and value autonomy, join Alignerr to influence how

Qualifications

  • 4+ years of professional software engineering experience.
  • Experience at a high-growth tech company or top-tier software organization.
  • Proficiency in Python and writing clean, well-tested code.

Responsibilities

  • Design and implement coding benchmarks for frontier AI models across real-world programming tasks.
  • Build scalable data pipelines for AI evaluation workflows.
  • Analyze AI-generated code for correctness, reliability, and edge-case failures.
  • Create evaluation scenarios that test reasoning, debugging, and code quality.
  • Work with large codebases and multi-language environments.
  • Collaborate on systems that improve AI model understanding and code generation.
  • Provide detailed technical feedback on model performance and failure patterns.
  • Contribute to evaluation framework designs that set industry standards.

Skills

Python proficiency
English communication
Autonomy
Code quality
Git proficiency

Education

CS/ML degree or equivalent experience

Tools

CI/CD pipelines
pytest
Mocha
JUnit
Git

Job description

About The Role

What if the code you write could determine how smart the next generation of AI truly is? We're looking for experienced Software Engineers to design and build the coding benchmarks and data pipelines used to evaluate frontier AI models - the systems that decide whether an AI can actually reason, debug, and write production-quality software.

About The Role

What if the code you write could determine how smart the next generation of AI truly is? We're looking for experienced Software Engineers to design and build the coding benchmarks and data pipelines used to evaluate frontier AI models - the systems that decide whether an AI can actually reason, debug, and write production-quality software.

This is high-impact, technically demanding work at the intersection of software engineering and AI research. You'll work with large codebases, multiple programming languages, and scalable infrastructure to create evaluation systems that push the boundaries of what AI can do.

This is a fully remote contract role. If you thrive in fast-paced engineering environments and want your work to directly shape the trajectory of AI - this is the role.

  • Organization: Alignerr
  • Type: Hourly Contract
  • Location: Remote
  • Contract Length: 3 Months
  • Commitment: Full-time availability preferred
What You'll Do
  • Design and implement coding benchmarks used to evaluate frontier AI models across real-world programming tasks
  • Build and maintain scalable data pipelines for AI evaluation workflows
  • Analyze AI-generated code for correctness, reliability, and edge-case failures
  • Create structured evaluation scenarios that rigorously test reasoning, debugging, and code quality
  • Work with large code repositories and multi-language environments
  • Collaborate on systems that improve how AI models understand and generate software
  • Provide detailed technical feedback on model performance and failure patterns
  • Contribute to the design of evaluation frameworks that set industry standards
Who You Are
  • 4+ years of professional software engineering experience - this is non-negotiable
  • Experience working at a high-growth tech company or top-tier software organization
  • Expert proficiency in Python - you write clean, performant, well-tested Python code
  • Hands-on experience with code repositories and working in large, complex codebases
  • Proven experience designing and implementing LLM coding benchmarks and data pipelines
  • Track record of working in high-performance engineering environments with large-scale products or platforms
  • Strong command of version control systems (Git) and modern development workflows
  • Bilingual or native English speaker with strong written communication skills
  • Self-directed, technically rigorous, and comfortable operating with autonomy
What Makes a Perfect Match
  • Senior or Lead-level engineering profiles with a history of technical ownership
  • Bachelor's or Master's degree in Computer Science, Machine Learning, or a related field - or equivalent professional experience
  • Proficiency in one or more additional languages: JavaScript, Go, C++, or other relevant languages
  • Experience with CI/CD pipelines and writing robust unit tests (pytest, Mocha, JUnit)
  • Background in security engineering or significant open-source contributions
  • Familiarity with AI/ML evaluation methodologies or model benchmarking
Why Join Us
  • Work on cutting-edge AI evaluation projects alongside world-class research teams
  • Fully remote - work from anywhere with a reliable internet connection
  • Your benchmarks directly influence how the most advanced AI systems in the world are measured and improved
  • Freelance autonomy with meaningful, high-stakes engineering work
  • Collaborate with a global community of elite engineers and researchers
  • Potential for contract extension and ongoing engagement as new evaluation challenges emerge
Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Software Engineer — AI Code Ranking
Software Engineer — AI Code Ranking

Alignerr • India

Remote
INR 3,246,000 - 7,790,000
Software Engineer (AI Training)
Software Engineer (AI Training)

Alignerr Corp. • Ahmedabad

Remote
INR 827,000 - 1,653,000
Software Engineer (AI Training)
Software Engineer (AI Training)

Alignerr Corp. • Delhi

Remote
INR 5,155,000 - 10,309,000
Fully remote
Flexible hours
Freelance autonomy
+2
Software Engineer (AI Training)
Software Engineer (AI Training)

Alignerr Corp. • Bengaluru

Remote
INR 1,933,000 - 5,799,000
Software Engineer (AI Training)
Software Engineer (AI Training)

Alignerr Corp. • Coimbatore

Remote
INR 689,000 - 3,444,000
Software Engineer (AI Training)
Software Engineer (AI Training)

Alignerr Corp. • Ahmedabad District

Remote
INR 1,378,000 - 2,480,000
Software Engineer (AI Training)
Software Engineer (AI Training)

Alignerr Corp. • Noida

Remote
INR 551,000 - 1,653,000
Software Engineer (AI Training)
Software Engineer (AI Training)

Alignerr Corp. • Gurugram District

Remote
INR 3,240,000 - 7,776,000
Fully remote
Flexible hours
Software Testing Analyst
Software Testing Analyst

Alignerr Corp. • India

Remote
INR 2,337,000 - 5,194,000
Fully remote
Flexible schedule
Freelance autonomy
Aubergine Solutions - Senior AI Evaluation Andamp; Reliability Engineer
Aubergine Solutions - Senior AI Evaluation Andamp; Reliability Engineer

Aubergine Solutions • Ahmedabad

On-site
INR 2,800,000 - 4,500,000