Data Annotator IT & Data Development

01.tech; Australia; Remote; AUD 65,000 - 95,000; Full time

Benefits offered by this job

Medical insurance

Annual benefits allowance

Professional training reimbursement

Relocation assistance

Corporate digital library

Psychologist sessions

Job summary

01.tech is seeking a data annotation specialist to annotate unstructured data across chats, reviews, transcripts, and images. You will create guidelines, label taxonomies, and drive quality with LLM-assisted labeling and human-in-the-loop processes.

You will coordinate annotators, manage external vendors, and oversee datasets through lifecycle, ensuring privacy compliance. Remote-friendly environment with opportunities for growth.

Qualifications

• 1.5-2+ years of experience in data annotation or preparing datasets for ML.
• Hands-on with unstructured data: text, audio, images.
• Experience with annotation tools: Label Studio, CVAT, Prodigy, Doccano.
• Experience managing crowdsourcing platforms and annotation vendors.
• Experience using LLMs for data annotation and quality evaluation.
• Knowledge of inter-annotator agreement metrics (kappa, alpha, precision/recall).
• Python scripting and automation, Pandas, JSON/CSV/JSONL, API calls.
• SQL skills for querying and aggregating data.
• English proficiency at B2 level or above.

Responsibilities

• Annotate unstructured data across support chats, inquiries, reviews, transcripts, and images.
• Develop annotation guidelines, label taxonomies, and edge-case instructions.
• Use LLMs for pre-labeling, automated quality evaluation, and synthetic data generation.
• Monitor annotation quality with inter-annotator metrics and gold sets.
• Coordinate annotators, calibration sessions, and provide feedback.
• Manage external annotation vendors and platforms, including SLAs and budgets.
• Manage datasets through lifecycle: versioning, splits, storage, documentation.
• Ensure data privacy and NDAs, access control for external vendors.
• Collaborate with Data Science and ML Eng teams to prioritize tasks.

Skills

Data annotation

NLP labeling

QA and metrics

Project coordination

Tools

Label Studio

CVAT

Prodigy

Doccano

Job description

• Annotate unstructured data, including support chats, customer inquiries, reviews, call recordings, transcripts, images, and documents. Perform classification, named entity recognition (NER), intent extraction, sentiment analysis, and evaluation of conversations and LLM-generated responses
• Develop annotation guidelines, label taxonomies, and instructions covering examples and edge cases. Configure annotation projects, create golden datasets, and design quality control tasks
• Use LLMs (GPT, Claude, Gemini, and open-source models) for pre-labeling, automated quality evaluation (LLM-as-a-judge), and synthetic data generation. Compare AI-generated labels with human annotations and implement human-in-the-loop workflows
• Monitor annotation quality using inter-annotator agreement metrics (Cohen's/Fleiss' kappa, Krippendorff's alpha, agreement rate), golden set accuracy, annotation speed, and cost efficiency
• Coordinate annotators, conduct calibration sessions, provide feedback, and evaluate individual performance
• Manage external annotation vendors and platforms (Toloka, Scale AI, Labelbox, Appen, or similar), including vendor selection, pilot projects, requirements, SLAs, budgets, deadlines, quality control, and deliverable acceptance
• Manage datasets throughout their lifecycle, including versioning, storage, deduplication, class balancing, train/validation/test splits, and documentation (data cards)
• Ensure compliance with data privacy requirements, including anonymization, NDAs, and access management for external vendors
• Collaborate with Data Science and ML Engineering teams to define data requirements, analyze model errors, and prioritize additional annotation tasks using active learning and hard examples

Requirements

• 1.5-2+ years of experience in data annotation or preparing datasets for machine learning, including unstructured data such as text, audio, and images
• Hands-on experience with annotation tools such as Label Studio, CVAT, Prodigy, Doccano, or similar platforms, including project setup, annotation templates, and data export
• Experience managing crowdsourcing platforms or annotation vendors, including task assignment, quality assurance, deliverable acceptance, and budget management
• Experience using LLMs for data annotation through web interfaces and APIs, including prompt engineering, few-shot prompting, structured JSON outputs, and quality evaluation
• Knowledge of annotation quality and inter-annotator agreement metrics (kappa, alpha, precision/recall on golden datasets), with the ability to interpret results and make quality-related decisions
• Working knowledge of Python for scripting and automation, including Pandas, JSON/CSV/JSONL processing, API calls, and basic data validation
• SQL skills for querying, filtering, and aggregating data
• Ability to write clear, structured annotation guidelines and technical instructions
• English proficiency at B2 level or above for documentation, communication with international vendors, and working with English-language datasets
• Nice to have experience preparing datasets for LLM training and evaluation, including SFT conversations, RLHF, preference data, red teaming, and AI assistant response evaluation; experience with data annotation in iGaming, betting, e-commerce, FinTech, or customer support environments; experience with speech transcription and annotation, including speaker diarization, ASR error analysis, and tools such as Whisper; understanding of fundamental ML concepts, including train/test splits, data leakage, and the impact of label noise on model performance; experience with active learning or weak supervision; experience annotating multilingual datasets or managing multilingual annotation teams
• The opportunity to work on large-scale products and contribute to their development
• A flexible work environment built on autonomy, ownership, and trust, without micromanagement
• The option to work fully remotely or from a modern office
• Medical insurance coverage in countries where the company operates
• An annual $2,000 benefits allowance after the probation period, which can be used for health insurance upgrades, language learning, and sports activities
• Partial reimbursement for professional training, courses, and participation in industry conferences
• Access to a corporate digital library
• Company-funded weekly online sessions with a psychologist
• Relocation opportunities to countries where the company has offices, with relocation assistance available after the probation period, subject to approval