Policy & Data Quality Specialist (Contract)

Blackhornvc; San Francisco (CA); Hybrid; USD 55,000 - 103,000; Part time

Job summary

Blackhornvc is seeking a Policy & Data Quality Specialist as a Contingent Worker (Contract) in the San Francisco area. You will train AI models to follow policies, ensure decision accuracy, create golden sets, calibrate vendor sites, and manage vendor relationships.

Experience in content moderation is valuable but not required. The role involves working with potentially objectionable content and collaborating with cross-functional teams to improve policies and processes.

Qualifications

• 4+ years of experience in technology consulting, content moderation, quality assurance, or a related field.
• Strong understanding of policy development, moderation procedures, and guidelines.
• Experience developing evals and golden sets.

Responsibilities

• Policy & Data Quality Assurance: review decisions for accuracy, consistency, and policy adherence.
• Golden Set Creation: develop and maintain golden sets to test models and vendor performance.
• Vendor Site Calibration: collaborate with vendor sites to identify improvements and calibrate plans.
• Vendor Management: maintain vendor relationships and provide guidance and performance metrics.
• Process and Policy Improvement: identify opportunities and implement changes with stakeholders.
• Data Analysis: analyze metrics to identify trends informing QA strategies.
• Communication: coordinate with operations, policy, engineering, and leadership teams.

Skills

Policy development

Data quality

Quality assurance

Content moderation

Golden sets

Vendor management

Data analysis

Communication

Tools

Data analysis tools

Job description

We are seeking a highly skilled and detail-oriented Policy & Data Quality Specialist to join our team as a Contingent Worker (Contract). The successful candidate will be responsible for training cutting edge AI models that strictly adhere to policies, for ensuring the quality and accuracy of decisions, creating golden sets, calibrating vendor sites, and managing vendor relationships. This is an excellent opportunity for someone who is passionate about policy analysis and quality assurance to make a meaningful impact on our organization. Experience in content moderation is valuable but not required. This role will involve exposure to potentially objectionable content including but not limited to graphic images, videos and writings, offensive or derogatory language, and other potential objectionable material i.e. child exploitation, graphic violence, animal abuse, self-injury and other content which may be considered offensive or disturbing.

Key Responsibilities

• Policy & Data Quality Assurance: Review and evaluate decisions made by expert models and vendors to ensure accuracy, consistency, and adherence to policies and guidelines.

• Golden Set Creation: Develop and maintain golden sets of content to test and calibrate expert models and vendor performance, ensuring that they meet our quality standards.

• Vendor Site Calibration: Collaborate with vendor sites to identify areas for improvement, provide feedback, and implement calibration plans to enhance their performance.

• Vendor Management: Build and maintain strong relationships with vendors, providing guidance, support, and performance metrics to ensure they meet our quality expectations.

• Process and Policy Improvement: Identify opportunities to improve policies and processes, workflows, and tools, and collaborate with internal stakeholders to implement changes.

• Data Analysis: Analyze data and metrics to identify trends, patterns, and insights that inform quality assurance strategies and vendor performance.

• Communication: Communicate effectively with internal stakeholders, including operations teams, policy teams, engineering teams and leadership, to ensure alignment and collaboration on quality assurance initiatives.

• Training and Development: Participate in training and development programs to stay up-to-date on policies, procedures, and industry best practices.

Requirements

• 4+ years of experience in technology consulting, content moderation, quality assurance, or a related field.

• Strong understanding of policy development, moderation procedures, and guidelines.

• Experience developing evals and golden sets.

• Familiarity with data analysis and reporting tools preferred.

• Strong attention to detail and ability to maintain accuracy in a fast-paced environment.

• Excellent analytical, problem-solving, and communication skills.

• Ability to work independently and collaboratively as part of a team.

• Experience working with vendors or external partners is valuable.

Nice to Have

• Experience with content moderation platforms and tools.

• Knowledge of industry best practices and standards for content moderation and quality assurance.

• Certification in content moderation, developing AI evals, or a related field.

Working Conditions:

• This will be a hybrid-role in an office setting.

• Must be able to work in a fast-paced environment with multiple priorities and deadlines.

• This role is expected to encounter and engage with objectionable content