Forward-Deployed QA Engineer - AI & LLM Testing (Remote)

Ellipsis Health, Inc.; San Francisco (CA); Hybrid; USD 120,000 - 140,000; Full time

Benefits offered by this job

401(k) matching

Health insurance

Vision insurance

Dental insurance

Flexible paid time off

Job summary

Ellipsis Health, Inc. is seeking a Forward Deployed QA Engineer based in San Francisco with the opportunity for remote work within the United States. The role focuses on ensuring the quality and stability of our conversational AI product while collaborating closely with engineering teams to troubleshoot and optimize workflows.

The ideal candidate will have a strong background in QA Engineering, expertise with LLM behaviors, and excellent communication skills for managing client stakeholders.

Qualifications

• Strong background in software quality assurance and end-to-end test strategies.
• Hands-on experience with LLM behaviors and call-tracing tools.
• Ability to write SQL queries and navigate cloud infrastructure logs.
• Fundamental understanding of Speech-to-Speech workflows.
• Basic familiarity with enterprise telephony routing.

Responsibilities

• Analyze client workflows to design comprehensive test cases.
• Build and execute automated test scenarios.
• Evaluate prompts used in the testing framework.
• Deploy various testing methodologies at the right timing.
• Collaborate with engineering teams to identify technical roadblocks.
• Perform daily root cause analysis on LLM failures.
• Investigate live customer calls and production incidents.
• Monitor the end-to-end voice AI pipeline.

Skills

QA Engineering

LLM Troubleshooting

SQL (PostgreSQL)

Speech-to-Speech Pipeline Knowledge

Telephony Routing Familiarity

Client Management

Job description

Ellipsis Health, Inc. is seeking a Forward Deployed QA Engineer based in San Francisco with the opportunity for remote work within the United States. The role focuses on ensuring the quality and stability of our conversational AI product while collaborating closely with engineering teams to troubleshoot and optimize workflows.

The ideal candidate will have a strong background in QA Engineering, expertise with LLM behaviors, and excellent communication skills for managing client stakeholders.