Remote-First Senior LLM Evaluation Architect

Remote; San Francisco (CA); Remote; USD 150,000 - 210,000; Full time

Benefits offered by this job

Remote-first

Competitive salary

Global career growth

Learning opportunities

Job summary

Binance is seeking an evaluation expert with an algorithmic background to build LLM evaluation capabilities across dialogue, trading, and other scenarios. You will quantify model performance, pinpoint issues, and guide R&D iterations using robust evaluation methods.

Responsibilities include designing end-to-end evaluation plans, building datasets with high coverage, and driving automation of evaluation pipelines to support rapid model iteration.

Qualifications

• Master's degree or above in CS/AI/Math/Stats with solid algorithmic foundation.
• Hands-on LLM evaluation experience at a large tech company.
• Familiar with evaluation methods and their boundaries; able to define dimensions and write discriminative rubrics.
• Systematic control over data representativeness, annotation consistency, and result reliability.
• Proficient in Python and automation of evaluation workflows.
• Strong business understanding and communication; able to drive cross-team collaboration.

Responsibilities

• Design end-to-end LLM evaluation plans for dialogue and financial trading.
• Lead the design and construction of evaluation datasets and benchmarks.
• Analyze model capability boundaries and provide actionable improvement recommendations.
• Drive automation and scaling of evaluation workflows and build evaluation platforms.
• Collaborate with algorithm, product, and data teams to translate objectives into clear evaluation standards.

Skills

Python

LLM evaluation

Experiment design

Data analysis

Communication

Education

Master's degree or above in CS/AI/Math/Stats

Tools

Evaluation platforms

Rubrics design

Data processing scripts

Job description

Binance is seeking an evaluation expert with an algorithmic background to build LLM evaluation capabilities across dialogue, trading, and other scenarios. You will quantify model performance, pinpoint issues, and guide R&D iterations using robust evaluation methods.

Responsibilities include designing end-to-end evaluation plans, building datasets with high coverage, and driving automation of evaluation pipelines to support rapid model iteration.