LLM QA Engineer - Automated Testing & Prompting

ICE; Atlanta (GA); On-site; USD 80,000 - 130,000; Full time

Job summary

ICE is seeking a QA Engineer to test LLM-based software, build and maintain automated tests, and help optimize prompts. You will join a team focused on delivering reliable AI-driven solutions and ensuring quality across multiple projects.

The role requires 3+ years in QA automation, strong Python skills, and experience with Agentic AI tools, with an emphasis on continuous learning and collaboration.

Qualifications

• 3+ years in QA and QA automation.
• Proficiency in Python programming.
• Experience with Agentic AI tools for test automation.
• Ability to learn quickly and adapt to new tech.
• Strong communication and teamwork in technical contexts.

Responsibilities

• Implement automated test frameworks and maintain existing ones for LLM-based apps.
• Create automated tests using AI to assess correctness of LLM outputs.
• Improve testing methodologies for LLM-driven applications.
• Contribute to designing and refining prompts used in LLM systems.

Skills

Python

QA automation

Agentic AI tools

Communication

Teamwork

Learning agility

Education

Bachelor’s degree in STEM

Job description

ICE is seeking a QA Engineer to test LLM-based software, build and maintain automated tests, and help optimize prompts. You will join a team focused on delivering reliable AI-driven solutions and ensuring quality across multiple projects.

The role requires 3+ years in QA automation, strong Python skills, and experience with Agentic AI tools, with an emphasis on continuous learning and collaboration.