Policy & Data Quality Specialist (Contract) — AI & Vendor Quality

Blackhornvc; San Francisco (CA); Hybrid; USD 55,000 - 103,000; Part time

Job summary

Blackhornvc is seeking a Policy & Data Quality Specialist as a Contingent Worker (Contract) in the San Francisco area. You will train AI models to follow policies, ensure decision accuracy, create golden sets, calibrate vendor sites, and manage vendor relationships.

Experience in content moderation is valuable but not required. The role involves working with potentially objectionable content and collaborating with cross-functional teams to improve policies and processes.

Qualifications

• 4+ years of experience in technology consulting, content moderation, quality assurance, or a related field.
• Strong understanding of policy development, moderation procedures, and guidelines.
• Experience developing evals and golden sets.

Responsibilities

• Policy & Data Quality Assurance: review decisions for accuracy, consistency, and policy adherence.
• Golden Set Creation: develop and maintain golden sets to test models and vendor performance.
• Vendor Site Calibration: collaborate with vendor sites to identify improvements and calibrate plans.
• Vendor Management: maintain vendor relationships and provide guidance and performance metrics.
• Process and Policy Improvement: identify opportunities and implement changes with stakeholders.
• Data Analysis: analyze metrics to identify trends informing QA strategies.
• Communication: coordinate with operations, policy, engineering, and leadership teams.

Skills

Policy development

Data quality

Quality assurance

Content moderation

Golden sets

Vendor management

Data analysis

Communication

Tools

Data analysis tools

Job description

Blackhornvc is seeking a Policy & Data Quality Specialist as a Contingent Worker (Contract) in the San Francisco area. You will train AI models to follow policies, ensure decision accuracy, create golden sets, calibrate vendor sites, and manage vendor relationships.

Experience in content moderation is valuable but not required. The role involves working with potentially objectionable content and collaborating with cross-functional teams to improve policies and processes.