Prompt Engineer & AI Evaluator

20byte; USA; Remote; USD 80.000 - 120.000; Vollzeit

Zusammenfassung

20byte is seeking a role to shape how enterprise LLMs behave at scale, blending prompt design with rigorous evaluation. You will iterate on system prompts for production features and build evaluation suites that measure accuracy, safety, and output consistency.

You will diagnose model failures, reproduce edge cases, document findings for engineering and product teams, and use Python scripting plus JSON/Markdown to automate tests and aggregate results.

Qualifikationen

• 1+ years hands-on experience with prompt engineering or AI evaluation in production.
• Fluent or native-level English (C1–C2) for prompting and evals.

Aufgaben

• Design, draft, and refine system prompts for production LLM features across multiple use cases.
• Build and run systematic evaluation suites that measure accuracy, safety, and output consistency over time.
• Investigate model failures, reproduce edge cases, and document findings for engineering and product teams.
• Apply version control and structured formats (JSON, Markdown) when iterating on prompts and eval datasets.
• Use basic Python scripting to automate test runs, parse model outputs, and aggregate results.

Kenntnisse

Prompt engineering

AI evaluation

English proficiency

Analytical thinking

Python scripting

JSON/Markdown familiarity

Tools

JSON

Markdown

Python

Jobbeschreibung

Role overview

This role focuses on shaping how enterprise large language models behave at scale, combining prompt design with rigorous evaluation work. The successful candidate will iterate on system prompts, design structured tests, and surface subtle failure modes such as hallucinations, unsafe outputs, and inconsistent formatting before they reach end users.

Responsibilities

• Design, draft, and refine system prompts for production LLM features across multiple use cases.
• Build and run systematic evaluation suites that measure accuracy, safety, and output consistency over time.
• Investigate model failures, reproduce edge cases, and document findings for engineering and product teams.
• Apply version control and structured formats (JSON, Markdown) when iterating on prompts and eval datasets.
• Use basic Python scripting to automate test runs, parse model outputs, and aggregate results.

Requirements

• 1+ years of hands‑on experience with prompt engineering or AI evaluation in a production environment.
• Fluent or native‑level English (C1–C2), since the majority of prompting and evals are conducted in English.
• Strong analytical and critical thinking skills, with the ability to spot subtle errors, bias, or hallucinations.
• Working familiarity with JSON, Markdown, and basic Python scripting for automation and data handling.
• Indonesian citizenship and current residence in Indonesia; the position is 100% full remote.

Nice to have

• Experience with LLM APIs, evaluation frameworks, or annotation tools used in NLP quality work.