LLM QA Engineer: Automation & Prompt Testing

Intercontinental Exchange Holdings, Inc.

Atlanta (GA)

On-site

USD 90,000 - 120,000

Full time

14 days+

Application generator

Turn this role into an interview — a resume and cover letter built around what this employer wants.

Get past ATS filters

Job summary

Intercontinental Exchange, Inc. is seeking a QA Engineer with experience testing LLM-based software. You will join the QA team to develop and execute automated tests for AI-enabled applications and contribute to crafting prompts that drive the behavior of these systems.

Candidates should have a STEM degree, 3+ years in QA, Python proficiency, and a track record of building test tools with AI capabilities. You will collaborate across engineering teams to ensure quality, reliability, and clear

Qualifications

Bachelor’s degree in STEM fields.
3+ years in QA and QA automation.
Experience using Agentic AI tools to build test automation tools.
Proficiency in Python.
Ability to learn quickly and adapt to new technologies and processes, with a focus on keeping up to date with the latest developments in LLM related technologies.
Ability to independently learn new complex business domains.
Strong problem-solving skills and attention to detail.
Excellent communication and teamwork skills.
Ability to communicate fluently and professionally in complex technical and business contexts.

Responsibilities

Implement automated test frameworks and take ownership of existing frameworks for LLM based applications.
Implement automated tests that use AI for reasoning on the correctness of LLM generated information.
Improve testing methodologies for LLM based applications.
Participate in designing and improving prompts used in LLM based applications.