Sr. Software Engineer - AI Evaluation and Knowledge

Super Micro Computer; San Jose (CA); Vor Ort; USD 160.000 - 200.000; Vollzeit

Zusammenfassung

Supermicro is seeking a Sr. Software Engineer to advance an AI evaluation and knowledge platform for handling AI Server and rack-related customer issues.

You will ensure agent outputs are accurate, actionable, and continuously improved through evaluation systems and a robust knowledge base. You will work with Domain SMEs to validate troubleshooting steps, design evaluation frameworks, and optimize prompts for higher first-time accuracy and usability.

Qualifikationen

• 2+ years of experience with LLM / Generative AI applications and evaluation.
• Experience in prompt engineering and reducing hallucinations.
• Experience systematically evaluating LLM or agent outputs.
• Experience with knowledge bases / RAG and retrieval pipelines.
• Strong analytical and cross-team communication skills.
• Ability to translate domain feedback into actionable improvements.
• Familiarity with Python to implement evaluation/data processing scripts.
• Experience in technical support or vertical AI agent applications is a plus.

Aufgaben

• Design and maintain the agent evaluation framework (test sets, automated scoring, human sampling review, regression testing).
• Own prompt engineering and structured output optimization to improve step accuracy.
• Manage Knowledge Base quality processes (content review, version control, retirement of outdated content, writing back new cases).
• Analyze agent errors and hallucinations, propose and implement improvements.
• Work with Domain SMEs to validate troubleshooting steps.
• Establish feedback loops from on-site results to agent improvements.
• Define and track success metrics (e.g., first-time accuracy, step usability).
• Support RAG retrieval quality tuning and knowledge presentation improvements.

Kenntnisse

LLM / Generative AI experience

Prompt engineering

Evaluation of LLM outputs

Cross-team collaboration

Data analysis

Python programming

Tools

Python

Jobbeschreibung

Select how often (in days) to receive an alert: Create Alert

Sr. Software Engineer - AI Evaluation and Knowledge

Location: San Jose, California, United States

About Supermicro:

Supermicro® is a Top Tier provider of advanced server, storage, and networking solutions for Data Center, Cloud Computing, Enterprise IT, Hadoop/ Big Data, Hyperscale, HPC and IoT/Embedded customers worldwide. We are the #5 fastest growing company among the Silicon Valley Top 50 technology firms. Our unprecedented global expansion has provided us with the opportunity to offer a large number of new positions to the technology community. We seek talented, passionate, and committed engineers, technologists, and business leaders to join us.

Job Summary:

We are building an AI Agent-powered platform to handle customer issues related to AI Servers and Racks. This role focuses on the quality of agent outputs, evaluation systems, and continuous improvement of the Knowledge Base, ensuring the agent produces accurate and actionable troubleshooting steps.

Essential Duties and Responsibilities:

• Design and maintain the agent evaluation framework (test sets, automated scoring, human sampling review, regression testing)
• Own Prompt Engineering and structured output optimization to improve step accuracy
• Manage Knowledge Base quality processes (content review, version control, retirement of outdated content, writing back new cases)
• Analyze agent errors and hallucinations, then propose and implement improvements
• Work closely with Domain SMEs to validate the correctness and practicality of troubleshooting steps
• Establish a feedback loop from on-site execution results back into agent improvements
• Help define success metrics (e.g., first-time accuracy rate, step usability rate) and track them continuously
• Support RAG retrieval quality tuning and knowledge presentation improvements

Qualifications:

• Experience in LLM / Generative AI applications, with practical experience in evaluation or quality (2+ years of experience is preferred)
• Strong familiarity with Prompt Engineering and practical methods to reduce hallucinations
• Experience systematically evaluating LLM or Agent outputs
• Experience with RAG or Knowledge Base implementation / management
• Strong analytical and cross-team communication skills
• Ability to turn domain expert feedback into actionable improvements
• Familiarity with Python and ability to implement basic evaluation or data processing scripts
• Experience in technical support, troubleshooting, or vertical AI Agent applications is a plus
• Experience with LLM-as-judge, automated evaluation pipelines, or A/B testing is a plus
• Experience in knowledge engineering, data annotation, or case library building is a plus
• Experience collaborating with non-AI domain experts is a plus

Salary Range

$160,000 - $ 200,000

The salary offered will depend on several factors, including your location, level, education, training, specific skills, years of experience, and comparison to other employees already in this role. In addition to a comprehensive benefits package, candidates may be eligible for other forms of compensation, such as participation in bonus and equity award programs.

EEO Statement

Supermicro is an Equal Opportunity Employer and embraces diversity in our employee population. It is the policy of Supermicro to provide equal opportunity to all qualified applicants and employees without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, age, disability, protected veteran status or special disabled veteran, marital status, pregnancy, genetic information, or any other legally protected status.