Senior AI Quality Engineer — LLMs & Evaluation Systems

Block; San Francisco (CA); On-site; USD 190,000 - 230,000; Full time

Job summary

Block is building AI-powered quality systems that evaluate product behavior at scale, using LLMs and AI agents to surface insights and guide product decisions.

We are seeking a Senior Machine Learning Engineer to lead the technical direction of next-generation quality platforms, define long-term architecture, and scale evaluation frameworks across engineering, product, and leadership teams.

Qualifications

• 5+ years of experience in software engineering, machine learning engineering or applied AI.
• Deep experience designing and shipping large-scale AI/ML systems in production environments.
• Strong expertise with LLMs, agents, evaluation systems, and retrieval architectures.
• Proven ability to lead ambiguous, high-impact technical initiatives across teams.
• Strong systems thinking and architectural judgment.
• Experience defining technical strategy and influencing roadmaps beyond your team.
• Excellent communication and cross-functional leadership skills.
• A track record of building platforms and frameworks for other engineers.

Responsibilities

• Lead the technical strategy and architecture for AI-driven quality and evaluation systems across products.
• Drive development of scalable systems using LLMs, agents, and behavioral signals to assess quality and surface insights.
• Define long-term approaches for evaluation and quality intelligence across product surfaces.
• Translate organizational needs into clear direction, roadmaps, and platform capabilities.
• Influence engineering standards for reliable, measurable AI systems.
• Lead cross-functional initiatives across product, infrastructure, data and AI teams.
• Mentor engineers through technical leadership and system thinking.
• Identify leverage opportunities where AI improves understanding and debugging of product behavior.

Skills

5+ years in software/ML engineering or

Large-scale AI/ML systems

LLMs and agents

Ambiguous high-impact leadership

Systems thinking

Technical strategy

Cross-functional leadership

Platform thinking

Tools

AI infrastructure

Job description

Block is building AI-powered quality systems that evaluate product behavior at scale, using LLMs and AI agents to surface insights and guide product decisions.

We are seeking a Senior Machine Learning Engineer to lead the technical direction of next-generation quality platforms, define long-term architecture, and scale evaluation frameworks across engineering, product, and leadership teams.