Senior Software Engineer – AI Evaluation

Alignerr

United States

Remote

USD 83,000 - 165,000

Part time

14 days+

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

Alignerr is seeking a Senior Software Engineer to design and scale evaluation infrastructure that measures AI performance. You will build tools, pipelines, and frameworks that help research teams understand model capabilities, failure modes, and how to improve systems.

The role emphasizes high-impact software engineering in a remote setting, requiring 4+ years of experience, strong Python skills, and cloud/CI/CD familiarity. This is an hourly contract with 20–40 hours per week.

Qualifications

  • 4+ years of professional software engineering experience.
  • Strong Python proficiency with data processing libraries.
  • Experience building scalable production systems and APIs.

Responsibilities

  • Design and build scalable evaluation pipelines and frameworks for AI model performance.
  • Develop automated testing harnesses, scoring systems, and benchmarking tools.
  • Write clean, production-quality code to process, analyze, and visualize evaluation datasets at scale.
  • Create and maintain APIs, dashboards, and internal tools for research teams to run evaluations.
  • Collaborate with AI researchers to translate evaluation methodologies into software.
  • Identify edge cases and reliability issues in AI outputs.
  • Optimize system performance and infrastructure costs.
  • Contribute to architecture and technical direction of the platform.

Skills

Python
Data processing (pandas/NumPy)
Web frameworks (FastAPI/Flask/Django)
Cloud / CI/CD familiarity

Tools

FastAPI
Flask
Django
Git
CI/CD

Job description

Senior Software Engineer – AI Evaluation
About the Role

What if your engineering skills could directly determine whether the world's most advanced AI systems are actually working? We're looking for Senior Software Engineers to design, build, and scale the evaluation infrastructure that measures AI performance — the critical layer between raw model output and real-world trust.

This is high-impact, technically challenging work at the intersection of software engineering and AI research. You'll build the tools, pipelines, and frameworks that help leading research teams understand what their models can do, where they fail, and how to make them better. If you love building robust systems and care deeply about quality and measurement, this role puts you at the center of the AI revolution.

  • Organization: Alignerr
  • Type: Hourly Contract
  • Location: Remote
  • Commitment: 20–40 hours/week

---

What You'll Do
  • Design and build scalable evaluation pipelines and frameworks for assessing AI model performance across diverse tasks and domains
  • Develop automated testing harnesses, scoring systems, and benchmarking tools for large language models and other AI systems
  • Write clean, production-quality code to process, analyze, and visualize evaluation datasets at scale
  • Create and maintain APIs, dashboards, and internal tools that enable research teams to run, track, and compare evaluations efficiently
  • Collaborate with AI researchers and data scientists to translate evaluation methodologies into reliable, repeatable software
  • Identify edge cases, failure modes, and reliability issues in AI outputs through systematic engineering approaches
  • Optimize system performance, data processing speed, and infrastructure costs
  • Contribute to the architecture and technical direction of the evaluation platform
  • Write clear documentation and participate in code reviews to maintain high engineering standards

---

Who You Are
  • 4+ years of professional software engineering experience, with a track record of building and shipping production systems
  • Strong proficiency in Python — including experience with data processing libraries (pandas, NumPy) and web frameworks (FastAPI, Flask, or Django)
  • Solid understanding of software architecture, design patterns, and engineering best practices
  • Experience working with large datasets and building data pipelines
  • Comfortable with cloud infrastructure (AWS, GCP, or Azure) and containerized deployments
  • Familiarity with version control (Git), CI/CD workflows, and testing frameworks
  • Strong problem-solving skills and the ability to work through ambiguity independently
  • Excellent written communication skills — you can document your work clearly and collaborate asynchronously
  • Self-motivated and reliable when working independently in a remote environment
Nice to Have
  • Experience with ML/AI evaluation, benchmarking, or model testing
  • Familiarity with LLMs, prompt engineering, or AI safety and alignment concepts
  • Background in building developer tools, internal platforms, or data infrastructure
  • Experience with distributed systems, message queues, or workflow orchestration (Airflow, Prefect, etc.)
  • Knowledge of statistical methods for measuring and comparing model performance
  • Prior experience in a remote-first or async-first engineering culture
  • Contributions to open-source projects related to AI, ML, or evaluation tooling

---

Why Join Us
  • Work on cutting-edge AI evaluation projects alongside world-class research labs
  • Directly influence how AI quality and safety are measured at scale — your code shapes the standard
  • Fully remote and flexible — work when and where you're most productive
  • Freelance autonomy with access to deeply meaningful, technically stimulating work
  • Collaborate with a global team of engineers and researchers pushing the boundaries of AI
  • Exposure to the latest developments in AI research, model capabilities, and evaluation science
  • Potential for ongoing work and contract extension as the platform and project scope grow
Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Python Insfrastructure Engineer - Model Evaluation
Python Insfrastructure Engineer - Model Evaluation

Alignerr Corp. • New York (NY)

Remote
USD 83,000 - 165,000
Python Insfrastructure Engineer - Model Evaluation
Python Insfrastructure Engineer - Model Evaluation

Alignerr • United States

Remote
USD 83,000 - 165,000
Senior Systems Engineer — AI Data & Infrastructure
Senior Systems Engineer — AI Data & Infrastructure

Alignerr • United States

Remote
USD 96,000 - 165,000
Fully remote
Lead Backend Engineer - AI Tooling
Lead Backend Engineer - AI Tooling

Alignerr Corp. • New York (NY)

Remote
USD 96,000 - 165,000
Software Engineer – Machine Learning (AI Training)
Software Engineer – Machine Learning (AI Training)

Alignerr Corp. • Seattle (WA)

Remote
USD 83,000 - 138,000
Software Engineer — AI Code Ranking
Software Engineer — AI Code Ranking

Alignerr • United States

Remote
USD 55,000 - 124,000
Systems Software Engineer - Machine Learning Ops
Systems Software Engineer - Machine Learning Ops

Alignerr Corp. • Seattle (WA)

Remote
USD 83,000 - 124,000
Systems Software Engineer - Machine Learning Ops
Systems Software Engineer - Machine Learning Ops

Alignerr • United States

Remote
USD 96,000 - 165,000
Remote work
AI Evaluation Engineer
AI Evaluation Engineer

DeepRec.ai • Denver (CO)

Remote
USD 162,000 - 198,000
Senior C# Full-Stack Engineer — AI Data & Infrastructure
Senior C# Full-Stack Engineer — AI Data & Infrastructure

Alignerr • United States

Remote
USD 103,000 - 152,000