Data Annotation Engineer

Programmers.io; Louisville (KY); On-site; USD 90,000 - 130,000; Full time

Job summary

Programmers.io is seeking an Annotation Program Lead to manage our AI quality annotation program onsite in Louisville. You will oversee the day-to-day operations, lead a team of annotators, and develop guidelines and quality standards for gold-standard datasets used to measure conversational AI performance and compliance.

The ideal candidate brings 5+ years in program management or related fields, strong stakeholder engagement skills, and proficiency with Python/R, SQL, and Excel.

Qualifications

• Bachelor's degree or equivalent experience in Linguistics/Psychology.
• 5+ years of experience in program management, operations, QA, data labeling, or related fields.
• Experience leading teams and managing operational workflows.
• Excellent written and verbal communication skills.
• Familiarity with Python/R, SQL and Excel. Running scripts on language data.
• Experience supporting AI, machine learning, or data annotation programs.
• Familiarity with Responsible AI, compliance, governance processes.

Responsibilities

• Lead the day-to-day operation of the AI annotation program.
• Manage a team of annotators responsible for reviewing customer conversations and AI interactions.
• Develop annotation guidelines, procedures, and quality standards.
• Establish calibration programs and QA processes to ensure consistency across reviewers.
• Partner with Data Scientists to create and maintain gold-standard datasets.
• Monitor annotation accuracy, throughput, and quality metrics.
• Coordinate reporting and deliverables for Legal, Compliance, Responsible AI, and executive stakeholders.
• Manage annotation workflows and tooling including setting up annotation jobs, running data processing scripts, and testing annotation UIs for quality before job launch.
• Plan capacity requirements to support seasonal increases in review volume.
• Identify process improvements that increase efficiency and annotation quality.

Skills

Program management

Quality assurance

Stakeholder management

Data labeling

Data analysis

Communication

Education

Bachelor's degree or equivalent experience in Linguistics/Psychology

Tools

Python

R

SQL

Excel

Job description

Location - Louisville, Kentucky ( Day 1 onsite)

Job Description:

We are looking for an Annotation Program Lead to manage our AI Quality Annotation Program. This individual will oversee the operational execution of human review activities that establish the gold-standard datasets used to measure conversational AI performance, safety, and compliance.

The ideal candidate combines strong program management skills with experience in quality assurance, data annotation operations, and stakeholder engagement.

• Lead the day-to-day operation of the AI annotation program.
• Manage a team of annotators responsible for reviewing customer conversations and AI interactions.
• Develop annotation guidelines, procedures, and quality standards.
• Establish calibration programs and quality assurance processes to ensure consistency across reviewers.
• Partner with Data Scientists to create and maintain gold-standard datasets.
• Monitor annotation accuracy, throughput, and quality metrics.
• Coordinate reporting and deliverables for Legal, Compliance, Responsible AI, and executive stakeholders.
• Manage annotation workflows and tooling including setting up annotation jobs, running data processing scripts, and testing annotation UIs for quality before job launch.
• Plan capacity requirements to support seasonal increases in review volume.

Identify process improvements that increase efficiency and annotation quality.

• Bachelor's degree or equivalent experience in Linguistics/Psychology.
• 5+ years of experience in program management, operations, quality assurance, data labeling, or related fields.
• Experience leading teams and managing operational workflows.
• Strong organizational and stakeholder management skills.
• Ability to analyze quality metrics and drive continuous improvement initiatives.
• Excellent written and verbal communication skills.
• Familiarity with Python/R, SQL and Excel. Should be familiar with running scripts on language data.
• Strong organizational and stakeholder management skills.
• Ability to analyze quality metrics and drive continuous improvement initiatives.
• Excellent written and verbal communication skills.
• Familiarity with Python/R, SQL and Excel. Should be familiar with running scripts on language data.
• Experience supporting AI, machine learning, or data annotation programs.
• Familiarity with Responsible AI, compliance, legal review, or governance processes.
• Experience developing operational standards and quality frameworks.
• Experience managing vendor or contractor resources.