Senior Python Engineer — AI Task Designer & Evaluator

Socket.dev; Town of Florida (NY); Hybrid; USD 165,000 - 276,000; Full time

Job summary

Mindrift is seeking a seasoned software engineer to join a project-based opportunity focused on building realistic AI evaluation environments. You will craft tasks, define success criteria, and write robust tests that accommodate multiple valid solutions.

The role emphasizes iteration and fairness in evaluation, with compensation tied to task acceptance and an up-to-$200/hr rate. The work is not permanent employment; you’ll estimate work according to task complexity and submit tasks by

Qualifications

• 8+ years of software development experience.
• Proficient with Python (FastAPI) and JavaScript/TypeScript (React).
• Hands-on experience with Docker, Postgres, Kafka, and Redis.
• Experience writing functional and integration tests.
• English proficiency at B2+ level.

Responsibilities

• Build realistic developer environments with codebases and context.
• Design tasks from intermediate states and define solvable criteria.
• Write tests that accept all valid solutions and reject incorrect ones.
• Iterate on tasks and tests based on QA feedback to ensure fairness.

Skills

8+ years dev

Python

FastAPI

JavaScript/TypeScript

React

Docker

Postgres

Kafka

Redis

Testing experience

English B2+

Tools

React

Job description

Mindrift is seeking a seasoned software engineer to join a project-based opportunity focused on building realistic AI evaluation environments. You will craft tasks, define success criteria, and write robust tests that accommodate multiple valid solutions.

The role emphasizes iteration and fairness in evaluation, with compensation tied to task acceptance and an up-to-$200/hr rate. The work is not permanent employment; you’ll estimate work according to task complexity and submit tasks by