AI Evaluation & Task Design Engineer

Biz Tech Consultants; United States; Remote; USD 120,000 - 180,000; Part time

Job summary

Biz Tech Consultants is seeking experienced software engineers to work on RLHF projects supporting the training and evaluation of AI coding agents. The role involves designing realistic technical tasks or reviewing and rating AI generated responses for accuracy and reliability, with emphasis on precise technical writing and high-quality review.

Responsibilities include creating tasks across software development, debugging, data processing, security, and ML; reviewing AI outputs for quality; and

Qualifications

• 4–7 years of experience building production systems, not just internal tools.
• Strong hands-on skill in Javascript.
• Comfortable with Docker and command line tools.
• Comfortable with one relational and one NoSQL database.
• Can point to a clear before/after metric from past work.
• Experience with remote or international clients or teams.
• Some exposure to secure data handling or compliance work.

Responsibilities

• Design technical tasks across software development, debugging, data processing, security, and ML, and set up environments and tests.
• Review AI generated responses for correctness, code quality, and adherence to instructions.
• Provide clear, evidence-based written feedback and comparative ratings.
• Validate task or evaluation difficulty against AI model outputs and revise based on feedback.

Skills

JavaScript

Python

Node.js

Go

Rust

Docker

Git

pytest

AI Integration

LLM

NoSQL

CI/CD

System Design

Tools

Kubernetes

Kafka

RabbitMQ

gRPC

Redis

Job description

Biz Tech Consultants is seeking experienced software engineers to work on RLHF projects supporting the training and evaluation of AI coding agents. The role involves designing realistic technical tasks or reviewing and rating AI generated responses for accuracy and reliability, with emphasis on precise technical writing and high-quality review.

Responsibilities include creating tasks across software development, debugging, data processing, security, and ML; reviewing AI outputs for quality; and