AI Content Engineer: Elevate LLM Experiences & Quality

Meta; Menlo Park (CA); Hybrid; USD 140,000 - 210,000; Full time

Job summary

Meta is seeking a Content Engineer to shape AI product experiences by defining and enforcing quality benchmarks across prompts, evaluations, and outputs. You will build scalable evaluation pipelines, translate user feedback into concrete metrics, and collaborate with research, engineering, product, and policy teams to improve model alignment and user satisfaction.

You will design test plans, manage annotators and golden sets, and drive practical improvements in production AI experiences,

Qualifications

• Bachelor's degree or equivalent experience with 5+ years in digital content strategy, user experience, technical writing, journalism, production, or related fields.
• Experience designing and running human evaluation pipelines at scale — including annotator management, rubric design, golden set construction, and calibration.
• Experience translating ambiguous feedback into structured, objective, fixable categories.
• Experience defining and operationalizing subjective quality dimensions into measurable benchmarks.
• Experience running structured software testing/QA programs — designing test plans, triaging results, and delivering actionable analysis to engineering teams fast enough to matter.
• Experience making editorial and content quality decisions in a fast-paced environment, communicating complex technical concepts to cross-functional partners, and leading through influence across teams without direct reporting lines.
• Experience with AI-native tooling and a bias toward using them to move faster.
• Experience with LLM-as-judge development — building automated quality signals aligned with human judgment, and validating alignment over time.
• Experience working with product teams or programs from roadmapping through delivery.
• Experience working in prompt engineering and agentic workflows.
• Demonstrated ability to integrate AI tools to optimize/redesign workflows and drive measurable impact.
• Experience adhering to responsible, ethical AI practices (risk assessment, bias mitigation, quality and accuracy reviews).
• Demonstrated ongoing AI skill development (prompt/context engineering, agent orchestration).

Responsibilities

• Own the quality bar for AI-powered product experiences and translate qualitative insights into measurable benchmarks.
• Build and operate evaluation infrastructure (human and automated) to measure model performance at product scale.
• Serve as a cross-functional bridge between research science, engineering, product, and policy to prioritize fixes.
• Define what great outputs look like and drive feedback loops to improve models and workflows.

Skills

Human evaluation pipelines

Ambiguity translation to structure

Quality benchmarks

Software testing/QA

Editorial and content quality

AI-native tooling

LLM-as-judge development

Cross-functional collaboration

Prompt engineering

Agent orchestration

AI ethics practices

AI skill development

Education

Bachelor's degree or equivalent experience

Tools

Annotation platforms

Prototyping environments

LLM-based development tools

Job description

Meta is seeking a Content Engineer to shape AI product experiences by defining and enforcing quality benchmarks across prompts, evaluations, and outputs. You will build scalable evaluation pipelines, translate user feedback into concrete metrics, and collaborate with research, engineering, product, and policy teams to improve model alignment and user satisfaction.

You will design test plans, manage annotators and golden sets, and drive practical improvements in production AI experiences,