Prompt Architect & AI Evaluation Specialist — Remote

20byte; United States; Remote; USD 80,000 - 120,000; Full time

Job summary

20byte is seeking a role to shape how enterprise LLMs behave at scale, blending prompt design with rigorous evaluation. You will iterate on system prompts for production features and build evaluation suites that measure accuracy, safety, and output consistency.

You will diagnose model failures, reproduce edge cases, document findings for engineering and product teams, and use Python scripting plus JSON/Markdown to automate tests and aggregate results.

Qualifications

• 1+ years hands-on experience with prompt engineering or AI evaluation in production.
• Fluent or native-level English (C1–C2) for prompting and evals.

Responsibilities

• Design, draft, and refine system prompts for production LLM features across multiple use cases.
• Build and run systematic evaluation suites that measure accuracy, safety, and output consistency over time.
• Investigate model failures, reproduce edge cases, and document findings for engineering and product teams.
• Apply version control and structured formats (JSON, Markdown) when iterating on prompts and eval datasets.
• Use basic Python scripting to automate test runs, parse model outputs, and aggregate results.

Skills

Prompt engineering

AI evaluation

English proficiency

Analytical thinking

Python scripting

JSON/Markdown familiarity

Tools

JSON

Markdown

Python

Job description

20byte is seeking a role to shape how enterprise LLMs behave at scale, blending prompt design with rigorous evaluation. You will iterate on system prompts for production features and build evaluation suites that measure accuracy, safety, and output consistency.

You will diagnose model failures, reproduce edge cases, document findings for engineering and product teams, and use Python scripting plus JSON/Markdown to automate tests and aggregate results.