Member Of Technical Staff — AI Systems & Benchmarking

Aionia Group; San Francisco (CA); On-site; USD 130,000 - 220,000; Full time

Benefits offered by this job

Equity

On-site

Job summary

AI Systems & Benchmarking in San Francisco is seeking a Member of Technical Staff to explore how AI systems perform in the real world and define how that performance is measured. You’ll shape evaluation frameworks, datasets, and metrics used by the industry, at the intersection of systems, analysis, AI, and strategy.

This is an on-site role with equity upside for early-stage impact. You’ll work with frontier models and real-world deployment contexts, translating complex behavior into actionable

Qualifications

• Must-have — strong Python proficiency with recent, hands-on production work.
• Experience with data analysis and building analytical frameworks.
• Ability to operate as a technical generalist across systems and domains.
• Clear, structured communication — translate complexity into insight.
• High ownership and comfort working in ambiguous environments.

Responsibilities

• Design and build AI evaluation and benchmarking systems.
• Analyze how models and agents perform across real-world use cases.
• Develop frameworks, datasets, and metrics to measure AI capabilities.
• Translate complex system behavior into clear, actionable insights.
• Work closely with engineers, product teams, and external partners.
• Contribute directly to product direction and overall strategy.

Skills

Python

Data analysis

Technical generalist

Clear communication

Ownership

Job description

AI Systems · Benchmarking · San Francisco

Member of Technical Staff

Most AI roles focus on building AI systems. This one focuses on something more fundamental — understanding how AI systems actually perform in the real world, and defining how that performance is measured. You’ll operate at the intersection of systems, analysis, AI, and strategy.

$130K – $220K + Equity On-Site · San Francisco AI Systems & Benchmarking Early-Stage Equity

This is not a typical AI role. You won’t just build AI systems — you’ll work on the problems that define how modern AI is evaluated, understood, and deployed. You’ll shape the frameworks, datasets, and metrics that the industry uses to make sense of model and agent behavior at scale.

“You’ll operate at the intersection of systems, analysis, AI, and strategy — on problems that most engineers never get to touch.”

Who This Is For

This Role Tends to Resonate With

This role attracts a specific type of person. You might be a fit if you:

Think deeply about how AI systems behave, not just how to build them

Have built real systems — APIs, pipelines, integrations, or products

Enjoy breaking down complex problems into structured, reusable frameworks

Are comfortable moving between technical detail and big-picture thinking

Have used AI tools in practice — LLMs, agents, and automated workflows

Still enjoy coding and working hands-on at the implementation level

Responsibilities

What You’ll Do

• Design and build AI evaluation and benchmarking systems
• Analyze how models and agents perform across real-world use cases
• Develop frameworks, datasets, and metrics to measure AI capabilities
• Translate complex system behavior into clear, actionable insights
• Work closely with engineers, product teams, and external partners
• Contribute directly to product direction and overall strategy

Requirements

What Matters Most

Must-Have — Non-Negotiable

• Strong Python proficiency with recent, hands-on production work
• Experience with data analysis and building analytical frameworks
• Ability to operate as a technical generalist across systems and domains
• Clear, structured communication — you translate complexity into insight
• High ownership and comfort operating in ambiguous environments

Backgrounds That Tend to Work Well

• Product-minded software engineers with meaningful AI exposure
• Engineers who’ve built systems involving LLMs, pipelines, or automation
• Technical professionals from top-tier consulting with real coding ability
• Founding or early engineers with broad, cross-functional ownership

The following profiles are unlikely to thrive in this role.

• Focused exclusively on training models or academic research
• Removed from hands-on coding and implementation work
• Purely management-focused without technical execution

Why This Role

Why This Role Is Different

Define how AI is measured, not just built

The work here shapes the frameworks and metrics the industry relies on to understand model and agent behavior — a rare, foundational problem space.

Direct exposure to cutting-edge AI systems

You’ll work with frontier models and real-world deployment contexts that most engineers don’t have visibility into.

High ownership in a small, fast-growing team

The team is expected to scale rapidly. Joining now means meaningful equity upside and the ability to shape how the function is built.

A rare blend of technical depth, strategy, and impact

This isn’t a pure engineering role or a pure strategy role. It’s both — for someone who can operate at that intersection.