LLM QA Engineer — Automated Testing & Prompting

Intercontinental Exchange (ICE); Atlanta (GA); On-site; USD 90,000 - 150,000; Full time

Job summary

Intercontinental Exchange, Inc. is seeking a QA Engineer to develop and maintain automated tests for LLM-based software. The role focuses on building and improving test frameworks and prompts, ensuring robust validation of AI-driven outputs.

Applicants should have a STEM degree and 3+ years in QA with Python expertise, strong problem-solving, and excellent communication. Located in Atlanta, this onsite role emphasizes collaboration and continuous learning.

Qualifications

• 3+ years in QA and QA automation.
• Experience using Agentic AI tools to build test automation tools.
• Proficiency in Python.
• Ability to learn quickly and adapt to new technologies and processes, with a focus on keeping up to date with the latest developments in LLM related technologies.
• Ability to independently learn new complex business domains.
• Strong problem-solving skills and attention to detail.
• Excellent communication and teamwork skills.
• Ability to communicate fluently and professionally in complex technical and business contexts.

Responsibilities

• Implement automated test frameworks and take ownership of existing frameworks for LLM based applications.
• Implement automated tests that use AI for reasoning on the correctness of LLM generated information.
• Improve testing methodologies for LLM based applications.
• Participate in designing and improving prompts used in LLM based applications.

Skills

Python

QA automation

LLM testing

AI test tools

Problem solving

Attention to detail

Communication

Teamwork

Learning agility

Adaptability

Education

Bachelor's degree in STEM fields

Job description

Intercontinental Exchange, Inc. is seeking a QA Engineer to develop and maintain automated tests for LLM-based software. The role focuses on building and improving test frameworks and prompts, ensuring robust validation of AI-driven outputs.

Applicants should have a STEM degree and 3+ years in QA with Python expertise, strong problem-solving, and excellent communication. Located in Atlanta, this onsite role emphasizes collaboration and continuous learning.