AI Automation Engineer

Fortinet; Sunnyvale (CA); On-site; USD 102,000 - 124,000; Full time

Benefits offered by this job

Medical insurance

Dental insurance

Vision insurance

Life and disability insurance

401(k) plan

11 paid holidays

Vacation time

Sick time

Comprehensive leave program

Job summary

Fortinet in Sunnyvale, CA is hiring a junior engineer to build and maintain our technical interview pipeline, from question bank to AI-assisted scoring and workflow automation. You will design systems to prompt and structure AI models, measure output against ground truth, catch errors, and grow the role into broader automation, working with the team to evolve the function.

The role requires Bachelor’s in CS or related field, strong programming fundamentals (Python, JS/TS), API experience, and

Qualifications

• Bachelor's degree in Computer Science or related field.
• Solid programming fundamentals (Python, JavaScript/TypeScript, or similar).
• Comfort working with APIs, including LLM/AI APIs (OpenAI, Anthropic, etc.).
• Demonstrated experience designing and evaluating prompts/pipelines for LLM-based tools (not just using AI products, but building with them).
• Proficient with Git and a hosted platform (GitHub or GitLab) - branching, PRs/MRs, code review, CI basics
• Basic understanding of databases (SQL or similar)
• Strong attention to detail and technical writing ability
• Ability to work independently on processes that are still being defined, and adapt as scope evolves

Responsibilities

• Architect scoring workflows that evaluate and compare candidate answers against reference answers/rubrics, grounded in a clear methodology for what 'correct' and 'well-scored' mean
• Calibrate and benchmark scoring prompts/models against human-graded samples, measuring accuracy and identifying systematic errors or bias
• Understand and account for model failure modes (inconsistency, hallucination, prompt sensitivity) and design safeguards around them
• Flag edge cases or low-confidence scores for human review rather than fully automating high-stakes decisions
• Automate the end-to-end workflow: question selection -> test delivery -> answer collection -> scoring -> reporting
• Help transition our source code and repositories to an internal Git platform
• Track test results to help establish scoring benchmarks over time
• Document the process, prompts, and scoring logic for auditability

Skills

Python

JavaScript/TypeScript

APIs (LLM/AI)

Prompt design

Git workflows

SQL basics

Technical writing

Independent work

Education

Bachelor's degree in Computer Science or related field

Tools

GitHub

GitLab

Job description

Job Description

We're looking for a junior engineer to be the primary engineer building and maintaining our technical interview pipeline - from the question bank to AI-assisted scoring to full workflow automation - working alongside the team to grow this role into a broader automation function over time. This isn't a "call an API and ship it" job: you'll be designing the systems that add/calibrate interview questions and evaluate candidate answers, which means understanding how to prompt and structure AI models for consistency, how to measure and validate output quality against ground truth, and how to catch and correct the ways these systems get things wrong. This is a role we expect to grow - the shape of that growth is still being defined.

Job Description

We're looking for a junior engineer to be the primary engineer building and maintaining our technical interview pipeline - from the question bank to AI-assisted scoring to full workflow automation - working alongside the team to grow this role into a broader automation function over time. This isn't a "call an API and ship it" job: you'll be designing the systems that add/calibrate interview questions and evaluate candidate answers, which means understanding how to prompt and structure AI models for consistency, how to measure and validate output quality against ground truth, and how to catch and correct the ways these systems get things wrong. This is a role we expect to grow - the shape of that growth is still being defined.

Responsibilities

We're looking for a junior engineer to be the primary engineer building and maintaining our technical interview pipeline - from the question bank to AI-assisted scoring to full workflow automation - working alongside the team to grow this role into a broader automation function over time. This isn't a "call an API and ship it" job: you'll be designing the systems that add/calibrate interview questions and evaluate candidate answers, which means understanding how to prompt and structure AI models for consistency, how to measure and validate output quality against ground truth, and how to catch and correct the ways these systems get things wrong. This is a role we expect to grow - the shape of that growth is still being defined.

Key Responsibilities

Assessment Repository Management

• Add, revise, and retire technical interview questions across roles/levels/skill areas
• Maintain question metadata (difficulty, topic tags, expected answer criteria, role relevance)
• Create variant/randomized versions of questions to reduce answer-sharing risk
• Build validation checks (automated and manual) to catch incorrect, ambiguous, or miscalibrated questions before they enter the repository
• Build automation to update problem sets and recalculate average difficulty ratings as new results come in
• Review and update questions periodically to prevent leakage/staleness and keep content aligned with actual job requirements

Scoring & Answer Evaluation

• Architect scoring workflows that evaluate and compare candidate answers against reference answers/rubrics, grounded in a clear methodology for what "correct" and "well-scored" mean
• Calibrate and benchmark scoring prompts/models against human-graded samples, measuring accuracy and identifying systematic errors or bias
• Understand and account for model failure modes (inconsistency, hallucination, prompt sensitivity) and design safeguards around them
• Flag edge cases or low-confidence scores for human review rather than fully automating high-stakes decisions

Process Automation

• Automate the end-to-end workflow: question selection -> test delivery -> answer collection -> scoring -> reporting
• Help transition our source code and repositories to an internal Git platform
• Track test results to help establish scoring benchmarks over time

Quality & Fairness

• Track scoring consistency and accuracy over time; report on pipeline health
• Watch for and mitigate AI evaluation bias across candidate demographics or answer styles
• Document the process, prompts, and scoring logic for auditability

Growth & Scope

• Apply the same automation and AI-integration skills developed here to other internal workflows as the role expands
• Partner with engineering teams to identify manual, repetitive processes that are good candidates for automation
• Take on additional internal tooling and infrastructure projects as capacity and scope grow

Required Qualifications

• Bachelor's degree in Computer Science or related field, or equivalent practical experience
• Solid programming fundamentals (Python, JavaScript/TypeScript, or similar)
• Comfort working with APIs, including LLM/AI APIs (OpenAI, Anthropic, etc.)
• Demonstrated experience designing and evaluating prompts/pipelines for LLM-based tools (not just using AI products, but building with them)
• Proficient with Git and a hosted platform (GitHub or GitLab) - branching, PRs/MRs, code review, CI basics
• Basic understanding of databases (SQL or similar)
• Strong attention to detail and technical writing ability
• Ability to work independently on processes that are still being defined, and adapt as scope evolves

Preferred Qualifications

• Familiarity with test/assessment platforms or ATS integrations
• Exposure to scripting automation (e.g., workflow tools, cron jobs, CI-style pipelines)
• Interest or coursework in fairness/bias in ML systems
• Prior experience conducting or coordinating technical interviews (not required, but a plus)

Must be authorized to work in the U.S. without sponsorship.

The US base salary range for this full-time position is $101,600-$124,200. Fortinet offers employees a variety of benefits, including medical, dental, vision, life and disability insurance, 401(k), 11 paid holidays, vacation time, and sick time, as well as a comprehensive leave program.

Wage ranges are based on various factors, including the labour market, job type, and job level. Exact salary offers will be determined by factors such as the candidate's subject knowledge, skill level, qualifications, experience, and geographic location.

All roles are eligible to participate in the Fortinet equity program. Bonus eligibility is reviewed at the time of hire and annually at the Company’s discretion.

Why Join Us

We encourage candidates from all backgrounds and identities to apply. We offer a supportive work environment and a competitive Total Rewards package to support you with your overall health and financial well-being.

Embark on a challenging, enjoyable, and rewarding career journey with Fortinet. Join us in bringing solutions that make a meaningful and lasting impact to our 890,000+ customers around the globe.

About Us

Fortinet (NASDAQ: FTNT) secures the largest enterprise, service provider, and government organizations around the world. Fortinet empowers its customers with intelligent, seamless protection across the expanding attack surface and the power to take on ever-increasing performance requirements of the borderless network - today and into the future. Only the Fortinet Security Fabric architecture can deliver security without compromise to address the most critical security challenges, whether in networked, application, cloud or mobile environments. Fortinet ranks number one in the most security appliances shipped worldwide and more than 500,000 customers trust Fortinet to protect their businesses.

We are committed to providing reasonable accommodations for all qualified individuals with disabilities. If you require assistance or accommodation due to a disability, please contact us at [e-mail].

Fortinet is an equal opportunity employer. We value diversity in our company, and all qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, disability, age, military/veteran status or any other applicable legally protected characteristics in the location in which the candidate is applying.