Lead, Product Content Engineering

Vamp; Los Angeles (CA); On-site; USD 193,000 - 269,000; Full time

Job summary

Meta seeks a Content Engineer to own the human evaluation layer for image and video generation, defining rubrics, running eval rounds, and translating subjective judgments into measurable signals.

The role partners with product, research, and modeling teams to set quality standards, determine when to automate, and drive model behavior decisions. A strong background in video production and evaluation frameworks is required.

Qualifications

• 10+ years of experience in video production, content strategy, editorial standards, media evaluation or relevant fields.
• Experience authoring evaluation rubrics for video and/or image quality, including dimension design, rating scales, tie-breaking rules and annotator guidelines
• Hands-on video production expertise, direction, shooting, editing, post and/or VFX, with the craft vocabulary to diagnose quality failures precisely
• Experience evaluating generative media output (text-to-video, image-to-video, video-to-video, text-to-image) and diagnosing model failure modes
• Experience partnering with AI/ML research or modeling teams, translating evaluation results into training, data or model-behavior recommendations
• Experience running human evaluation programs at scale: rater calibration, inter-rater reliability, golden sets, quality audits
• Experience leveraging quantitative insights to inform content and quality decisions
• Proven ability to build influence and drive alignment across product, engineering, design, research and analytics partners
• Experience leading and/or mentoring others
• Bachelor's degree in a related field, or equivalent practical experience

Responsibilities

• Own the evaluation rubrics for image and video generation models, define the quality dimensions, scales and decision rules, and maintain them as models and capabilities evolve.
• Design and run human eval rounds that produce training and fine-tuning signal, including prompt set construction, golden sets and side-by-side model comparisons.
• Set and defend the launch quality bar for generative media features, and make clear go / no-go quality recommendations.
• Calibrate and manage rater pools and vendor teams
• monitor inter-rater reliability and drive measurable improvements in agreement.
• Partner with research and modeling teams to translate eval findings into model, data and post-training priorities.
• Scale human judgment into automated measurement (LLM-as-a-judge, auto-metrics) and validate automated scores against human ground truth.
• Lead through collaboration, mentoring other content engineers on eval craft and rubric design.
• Stay ahead of the generative media landscape, benchmarking against competitive and open-source model output.
• Deliver high-quality evaluation work on model and launch timelines in a fast-paced, dynamic environment.

Skills

Ten+ years experience

Rubrics design

Video production

Generative media evaluation

Partner with ML/AI teams

Large-scale eval programs

Quantitative insight

Mentoring others

Education

Bachelor's degree in a related field

Equivalent practical experience

Tools

Annotation platforms

Eval tooling

Data pipelines

Job description

Product Content Engineering is a horizontal function supporting initiatives across Instagram and Meta AI, and is seeking a content engineer to lead media generation evaluations. We partner closely with product, research and modeling teams to set standards of quality and define what "good" looks like for generative image and video experiences. This position is focused on model training for image and video generation capabilities. You will own the human evaluation layer that tells our models what quality means, designing the rubrics, running the eval rounds, and turning subjective creative judgment into reproducible, measurable signals that directly shapes model behavior and launch decisions. The ideal candidate has built video and image evaluation rubrics from scratch and has hands-on video production expertise - they can articulate why a generated clip fails on motion coherence, lighting continuity, subject consistency, pacing, framing or audio sync, and can write that judgment down so that dozens of raters apply it the same way. They are fluent in text-to-video, image-to-video and video-to-video generation, and in the failure modes specific to each. You will operate as the quality authority across multiple modeling workstreams: setting the taxonomy, arbitrating disagreement, calibrating rater pools, and translating eval results into clear guidance for researchers and product leaders. You will also drive where human evaluation should give way to automated or LLM-as-a-judge measurement, and prove out that transition with data.

Lead, Product Content Engineering Responsibilities:

• Own the evaluation rubrics for image and video generation models, define the quality dimensions, scales and decision rules, and maintain them as models and capabilities evolve.
• Design and run human eval rounds that produce training and fine-tuning signal, including prompt set construction, golden sets and side-by-side model comparisons.
• Set and defend the launch quality bar for generative media features, and make clear go / no-go quality recommendations.
• Calibrate and manage rater pools and vendor teams
• monitor inter-rater reliability and drive measurable improvements in agreement.
• Partner with research and modeling teams to translate eval findings into model, data and post-training priorities.
• Scale human judgment into automated measurement (LLM-as-a-judge, auto-metrics) and validate automated scores against human ground truth.
• Lead through collaboration, mentoring other content engineers on eval craft and rubric design.
• Stay ahead of the generative media landscape, benchmarking against competitive and open-source model output.
• Deliver high-quality evaluation work on model and launch timelines in a fast-paced, dynamic environment.

Minimum Qualifications:

• 10+ years of experience in video production, content strategy, editorial standards, media evaluation or relevant fields
• Experience authoring evaluation rubrics for video and/or image quality, including dimension design, rating scales, tie-breaking rules and annotator guidelines
• Hands-on video production expertise, direction, shooting, editing, post and/or VFX, with the craft vocabulary to diagnose quality failures precisely
• Experience evaluating generative media output (text-to-video, image-to-video, video-to-video, text-to-image) and diagnosing model failure modes
• Experience partnering with AI/ML research or modeling teams, translating evaluation results into training, data or model-behavior recommendations
• Experience running human evaluation programs at scale: rater calibration, inter-rater reliability, golden sets, quality audits
• Experience leveraging quantitative insights to inform content and quality decisions
• Proven ability to build influence and drive alignment across product, engineering, design, research and analytics partners
• Experience leading and/or mentoring others
• Bachelor's degree in a related field, or equivalent practical experience

Preferred Qualifications:

• Experience building an evaluation function or quality framework from zero-to-one
• Background in film, television, OTT, commercial or VFX production
• Familiarity with generative media tooling and model families, and with prompt engineering for media generation
• Experience with annotation platforms, eval tooling and data pipelines supporting model training
• Experience working with product teams or programs from roadmapping through delivery
• Demonstrated ability to integrate AI tools to optimize/redesign workflows and drive measurable impact (e.g., efficiency gains, quality improvements)
• Experience adhering to and implementing responsible, ethical AI practices (e.g., risk assessment, bias mitigation, quality and accuracy reviews)
• Demonstrated ongoing AI skill development (e.g., prompt/context engineering, agent orchestration) and staying current with emerging AI technologies
• Experience working independently and navigating ambiguity in fast-paced environments

About Meta:

Meta builds technologies that help people connect, find communities, and grow businesses. When Facebook launched in 2004, it changed the way people connect. Apps like Messenger, Instagram and WhatsApp further empowered billions around the world. Now, Meta is moving beyond 2D screens toward immersive experiences like augmented and virtual reality to help build the next evolution in social technology. People who choose to build their careers by building with us at Meta help shape a future that will take us beyond what digital connection makes possible today - beyond the constraints of screens, the limits of distance, and even the rules of physics.

Meta is proud to be an Equal Employment Opportunity and Affiliation employer. We do not discriminate based upon race, religion, color, national origin, sex (including pregnancy, childbirth, or related medical conditions), sexual orientation, gender, gender identity, gender expression, transgender status, sexual stereotypes, age, status as a protected veteran, status as an individual with a disability, or other applicable legally protected characteristics. We also consider qualified applicants with criminal histories, consistent with applicable federal, state and local law. Meta participates in the E-Verify program in certain locations, as required by law. Please note that Meta may leverage artificial intelligence and machine learning technologies in connection with applications for employment.

Meta is committed to providing reasonable accommodations for candidates with disabilities in our recruiting process. If you need any assistance or accommodations due to a disability, please let us know at [e-mail].

$193,000/year to $269,000/year + bonus + equity + benefits

Individual compensation is determined by skills, qualifications, experience, and location. Compensation details listed in this posting reflect the base hourly rate, monthly rate, or annual salary only, and do not include bonus, equity or sales incentives, if applicable. In addition to base compensation, Meta offers benefits. Learn more about benefits at Meta.