Remote IT & AI Systems Evaluation Specialist

agency; United States; Remote; USD 8,300 - 90,000; Part time

Benefits offered by this job

Remote work

Freelance contract

Job summary

Agency seeks a detail-oriented evaluator to assess language models' ability to reason about computer information systems. You will apply IT and systems expertise to practical scenarios, verifying technical accuracy and sound logic.

In this remote freelance role, you will discuss IT situations, test architecture topics, capture error traces, and recommend prompt and evaluation-method improvements.

Qualifications

• Strong knowledge of computer information systems, IT, or related technical area.
• Clear, analytical communication showing how conclusions were reached.
• Bachelor’s or Master’s degree preferred in related field.

Responsibilities

• Discuss real-world IT situations and theoretical systems architecture questions with a language model.
• Evaluate responses for technical accuracy and logical consistency.
• Test subjects including software architecture, systems analysis, data management, cloud infrastructure, cybersecurity, database design, and IT project management.
• Probe enterprise integration, information governance, network configuration, security frameworks, distributed computing, SDLC, and IT compliance.
• Capture reproducible error traces and recommend changes to prompts and evaluation methods.

Skills

Analytical thinking

Clear communication

IT knowledge

Education

Bachelor’s or Master’s degree in computer information systems/IT/CS

Job description

Agency seeks a detail-oriented evaluator to assess language models' ability to reason about computer information systems. You will apply IT and systems expertise to practical scenarios, verifying technical accuracy and sound logic.

In this remote freelance role, you will discuss IT situations, test architecture topics, capture error traces, and recommend prompt and evaluation-method improvements.