QA Engineer: AI & LLM Testing Automation

careers-ice; Atlanta (GA); On-site; USD 90,000 - 150,000; Full time

Job summary

ICE Mortgage Technology is assembling a world-class Quality Engineering organization to advance automated testing for LLM-based collections and loss mitigation tools. You will implement and own QA frameworks, applying AI-driven reasoning to validate LLM outputs while driving testing methodology improvements.

The role requires a Bachelor’s in STEM and 3+ years in QA/QA automation, with strong Python skills and experience with agentic AI tools. Collaboration and clear communication are essential.

Qualifications

• Bachelor’s degree in STEM fields.
• 3+ years in QA and QA automation.
• Experience using Agentic AI tools to build test automation tools.
• Proficiency in Python.
• Ability to learn quickly and adapt to new technologies with focus on LLM developments.
• Ability to independently learn new complex business domains.
• Strong problem-solving skills and attention to detail.
• Excellent communication and teamwork skills.
• Ability to communicate fluently and professionally in complex technical and business contexts.

Responsibilities

• Implement automated test frameworks and take ownership of existing frameworks for LLM based applications.
• Implement automated tests that use AI for reasoning on the correctness of LLM generated information.
• Improve testing methodologies for LLM based applications.
• Participate and design and improve prompts used in LLM based applications.

Skills

Python

QA automation

Agentic AI tools

Learning agility

Attention to detail

Communication

Teamwork

Education

Bachelor’s degree in STEM

Job description

ICE Mortgage Technology is assembling a world-class Quality Engineering organization to advance automated testing for LLM-based collections and loss mitigation tools. You will implement and own QA frameworks, applying AI-driven reasoning to validate LLM outputs while driving testing methodology improvements.

The role requires a Bachelor’s in STEM and 3+ years in QA/QA automation, with strong Python skills and experience with agentic AI tools. Collaboration and clear communication are essential.