Software Engineer, Agent Harness & Evaluation

SIVAD Consulting; San Jose (CA); On-site; USD 180,000 - 325,000; Full time

Job summary

SIVAD Consulting is hiring a Software Engineer for Agent Harness & Evaluation in San Jose, CA. You will improve AI agents, design evaluations, and translate experiments into production software.

You will work with a cross-disciplinary team of engineers, researchers, and domain experts to deliver reliable agent solutions for hardware applications, with a focus on performance and customer outcomes.

Qualifications

• Strong software engineering skills in Python and TypeScript.
• Familiarity with modern language models, agent systems, and ML workflows.
• Strong communication skills and customer judgment.
• Self-directed with ownership and product judgment.
• Experience with LLM agents, orchestration, tools, memory, tracing, ML inference, and transformer architectures.

Responsibilities

• Find failure modes of AI agents and design evaluations to improve performance, reliability, and efficiency.
• Build new agent capabilities, including tool interfaces, context management, memory, and multi-agent orchestration.
• Analyze traces and results to identify genuine improvements over noise.
• Collaborate with customers and domain experts to understand needs and ship fixes within ~3 months.
• Turn successful experiments into well-built production software.

Skills

Python

TypeScript

LLM agents

Agent orchestration

ML workflows

Communication

Customer facing

Ownership

Large-scale systems

Education

Bachelor's in CS/EE

Tools

Tracing

Experiment tooling

Job description

Software Engineer, Agent Harness & Evaluation – San Jose, CA

$180k to $325k

We are hiring Software Engineers to improve the performance, reliability, and efficiency of AI agents for hardware applications. You'll identify failure modes, design and run evaluations, develop new agent capabilities, and turn successful experiments into production software. You'll join a sizable, rapidly growing team of software engineers, researchers, hardware experts, customers, and founding and field teams, working across customer discovery, evaluation design, applied research, product development, and deployment for a broad range of hardware-engineering problems.

Responsibilities

• Finding where AI agents fail, then designing and running evaluations to improve their performance, reliability, and efficiency
• Building new agent capabilities, including tool interfaces, context management, memory, and multi-step or multi-agent orchestration
• Digging into traces and experiment results to find bottlenecks, and telling real improvements apart from benchmark noise
• Working with customers and domain experts to understand what they need, then shipping and maintaining fixes from start to finish (within about 3 months)
• Turning successful experiments into well-built, maintainable production software

Qualifications

• Strong software engineering skills, particularly in Python and TypeScript.
• Familiarity with modern language models, agent systems, and common ML workflows.
• Strong communication skills, customer judgment, and the ability to identify the crux of ambiguous technical problems.
• Self-directed and ambitious, with strong ownership, creativity, and product judgment.
• Strong commitment to customer and business outcomes.
• Experience with LLM agents, orchestration, tools, memory, tracing, ML inference, and transformer architectures
• Ability to interface directly with customers to understand requirements and deliver technical solutions
• 3 - 15 years of experience as a software engineer with work on large-scale agent systems
• Experience at a rigorous tech company (Applied Intuition, Palantir, Tesla, etc.) or AI startup (Fireworks, Harvey, Y Combinator companies, etc.)

Nice to Haves

• Interest in EDA, RTL, or hardware-software systems.
• Experience or interest in electronic design automation (EDA), semiconductor domains, RTL, or hardware-software systems

Education

• Bachelor's in CS, EE, or similar degree from a top university