Senior AI Benchmarking & Systems Architect

Aionia Group; San Francisco (CA); On-site; USD 130,000 - 220,000; Full time

Benefits offered by this job

Equity

On-site

Job summary

AI Systems & Benchmarking in San Francisco is seeking a Member of Technical Staff to explore how AI systems perform in the real world and define how that performance is measured. You’ll shape evaluation frameworks, datasets, and metrics used by the industry, at the intersection of systems, analysis, AI, and strategy.

This is an on-site role with equity upside for early-stage impact. You’ll work with frontier models and real-world deployment contexts, translating complex behavior into actionable

Qualifications

• Must-have — strong Python proficiency with recent, hands-on production work.
• Experience with data analysis and building analytical frameworks.
• Ability to operate as a technical generalist across systems and domains.
• Clear, structured communication — translate complexity into insight.
• High ownership and comfort working in ambiguous environments.

Responsibilities

• Design and build AI evaluation and benchmarking systems.
• Analyze how models and agents perform across real-world use cases.
• Develop frameworks, datasets, and metrics to measure AI capabilities.
• Translate complex system behavior into clear, actionable insights.
• Work closely with engineers, product teams, and external partners.
• Contribute directly to product direction and overall strategy.

Skills

Python

Data analysis

Technical generalist

Clear communication

Ownership

Job description

AI Systems & Benchmarking in San Francisco is seeking a Member of Technical Staff to explore how AI systems perform in the real world and define how that performance is measured. You’ll shape evaluation frameworks, datasets, and metrics used by the industry, at the intersection of systems, analysis, AI, and strategy.

This is an on-site role with equity upside for early-stage impact. You’ll work with frontier models and real-world deployment contexts, translating complex behavior into actionable