Python Insfrastructure Engineer - Model Evaluation

Alignerr

United States

Remote

USD 83,000 - 165,000

Part time

14 days+

A complete application in a minute

Tailored resume and cover letter, ready to send.

Get past ATS filters

Job summary

Alignerr is seeking a Senior Python Infrastructure Engineer to design and build data pipelines, annotation tooling, and evaluation systems at the heart of AI research. This fully remote, hourly contract role offers flexible hours and collaboration with leading AI labs.

You will work with cross-functional teams to support model training and evaluation workflows, improve system reliability, and instrument observability.

Qualifications

  • 3–5+ years of production-grade Python experience.
  • Experience building evaluation harnesses for ML models.
  • Strong observability, metrics collection, and reliability skills.

Responsibilities

  • Design, build, and optimize high-performance Python systems for data pipelines and model evaluation.
  • Develop full-stack tooling and backend services for large-scale data annotation and validation.
  • Build evaluation harnesses for ML models and integrate with inference frameworks.
  • Improve reliability, performance, and safety of existing Python codebases.
  • Instrument observability tooling and metrics to monitor model performance.
  • Identify bottlenecks and design scalable solutions.
  • Collaborate with data, research, and engineering teams on training/evaluation workflows.
  • Participate in design reviews to iteratively improve architecture.

Skills

Python
Observability
Full-stack development
Data pipelines
Infrastructure tooling

Tools

PyTorch
TensorFlow

Job description

Python Infrastructure Engineer - Model Evaluation (AI Training)
About the Role

What if your Python engineering skills could directly shape how the world's most advanced AI models are built, tested, and improved? We're looking for a Senior Python Infrastructure Engineer to design and build the data pipelines, annotation tooling, and evaluation systems that sit at the heart of cutting-edge AI research.

This is a fully remote, flexible contract role working alongside leading AI labs on real production infrastructure. If you're a seasoned Python engineer who wants meaningful, high-impact work - this is it.

  • Organization: Alignerr
  • Type: Hourly Contract
  • Location: Remote
  • Commitment: 20-40 hours/week
What You'll Do
  • Design, build, and optimize high-performance Python systems supporting AI data pipelines and model evaluation workflows
  • Develop full-stack tooling and backend services for large-scale data annotation, validation, and quality control
  • Build and maintain evaluation harnesses for ML models, integrating with inference frameworks
  • Improve reliability, performance, and safety across existing Python codebases
  • Instrument systems with observability tooling and metrics collection to monitor reliability and model performance
  • Identify bottlenecks and edge cases in data and system behavior, and implement scalable solutions
  • Collaborate with data, research, and engineering teams to support model training and evaluation workflows
  • Participate in synchronous design reviews to iterate on architecture and implementation decisions
Who You Are
  • Native or fluent English speaker with clear written and verbal communication skills
  • Full-stack developer with a strong systems programming background
  • 3-5+ years of professional experience writing production-grade Python
  • Experienced building evaluation harnesses for ML models and integrating with inference frameworks
  • Solid background in observability, metrics collection, and monitoring system reliability
  • Self-directed and dependable - able to commit 20-40 hours per week and deliver consistently
Nice to Have
  • Prior experience with data annotation, data quality pipelines, or evaluation systems
  • Familiarity with AI/ML workflows, model training, or benchmarking pipelines
  • Experience with distributed systems or developer tooling
  • Background working directly with AI research teams or at AI-focused companies
Why Join Us
  • Work on production systems powering some of the most advanced AI research in the world
  • Fully remote and async-friendly - work from wherever you do your best thinking
  • Freelance autonomy with the depth and structure of meaningful, long-term engineering work
  • Collaborate directly with AI researchers and engineers at leading labs
  • Potential for ongoing work and contract extension as new projects launch
Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Python Insfrastructure Engineer - Model Evaluation
Python Insfrastructure Engineer - Model Evaluation

Alignerr Corp. • New York (NY)

Remote
USD 83,000 - 165,000
Principal Python Engineer - ML Infrastructure
Principal Python Engineer - ML Infrastructure

Alignerr • United States

Remote
USD 83,000 - 165,000
Senior Python Full-Stack Engineer — AI Data & Infrastructure
Senior Python Full-Stack Engineer — AI Data & Infrastructure

Alignerr • United States

Remote
USD 96,000 - 152,000
Python Software Engineer - AI Workflows
Python Software Engineer - AI Workflows

Alignerr Corp. • Seattle (WA)

Remote
USD 69,000 - 124,000
Fully remote
Contract-based freelance work
Flexible hours
+1
Python Software Engineer - AI Workflows
Python Software Engineer - AI Workflows

Alignerr • United States

Remote
USD 83,000 - 124,000
Data Platform Engineer (Python)
Data Platform Engineer (Python)

Alignerr • United States

Remote
USD 83,000 - 138,000
Lead Backend Engineer - AI Tooling
Lead Backend Engineer - AI Tooling

Alignerr Corp. • New York (NY)

Remote
USD 96,000 - 165,000
Python Engineer
Python Engineer

Alignerr • United States

Remote
USD 55,000 - 110,000
Data Platform Engineer (Python)
Data Platform Engineer (Python)

Alignerr Corp. • New York (NY)

Remote
USD 96,000 - 152,000
Senior Software Engineer – AI Evaluation
Senior Software Engineer – AI Evaluation

Alignerr • United States

Remote
USD 83,000 - 165,000