RLHF AI Evaluator & Task Designer (Python)

Biz Tech Consultants; United States; Remote; USD 55,000 - 110,000; Part time

Job summary

Biz Tech Consultants is seeking experienced software engineers to work on RLHF projects that support the training and evaluation of AI coding agents. You will design realistic technical tasks and review AI-generated responses for correctness and clarity, providing evidence-based feedback.

Ideal candidates have 4–7 years building production systems, strong JavaScript skills, and hands-on experience with Docker, databases, and remote or international teams.

Qualifications

• 4 to 7 years of experience building production systems.
• Strong hands-on skill in Javascript.
• Comfortable with Docker and command line tools.
• Experience with one relational and one NoSQL database.
• Evidence of a clear before/after metric from past work.
• Experience working with remote or international teams.
• Exposure to secure data handling or compliance work.

Responsibilities

• Design technical tasks across software development, debugging, data processing, security and ML; set up environments and tests as needed.
• Review AI-generated responses for correctness, code quality and adherence to instructions.
• Provide clear, evidence-based written feedback and comparative ratings across responses.
• Validate task or evaluation difficulty against AI model outputs and revise as needed.

Skills

Technical writing

Code review

Task authoring

Remote collaboration

Tools

JavaScript

Node.js

Python

Go

Rust

Docker

Git

pytest

LLM API

Kubernetes

Kafka

Redis

gRPC

CI/CD

System Design

Job description

Biz Tech Consultants is seeking experienced software engineers to work on RLHF projects that support the training and evaluation of AI coding agents. You will design realistic technical tasks and review AI-generated responses for correctness and clarity, providing evidence-based feedback.

Ideal candidates have 4–7 years building production systems, strong JavaScript skills, and hands-on experience with Docker, databases, and remote or international teams.