QA Automation Engineer, AI & LLM Testing

ICE; Atlanta (GA); On-site; USD 90,000 - 130,000; Full time

Job summary

ICE Mortgage Technology is expanding its Quality Engineering capabilities in Atlanta to deliver a best-in-class platform for collections, loss mitigation, and homeowner support. This role emphasizes building automated test frameworks for LLM-based applications, AI-driven validation, and prompt design, with a strong focus on Python and scalable QA practices.

You will contribute to innovative testing strategies, collaborate with cross-functional teams, and stay current with emerging LLM

Qualifications

• Bachelor’s degree in STEM fields.
• 3+ years in QA and QA automation.
• Experience using Agentic AI tools to build test automation tools.
• Proficiency in Python.
• Ability to learn quickly and adapt to new technologies and processes, with a focus on keeping up to date with the latest developments in LLM related technologies.
• Ability to independently learn new complex business domains.
• Strong problem-solving skills and attention to detail.
• Excellent communication and teamwork skills.
• Ability to communicate fluently and professionally in complex technical and business contexts.

Responsibilities

• Implement automated test frameworks and take ownership of existing frameworks for LLM based applications.
• Implement automated tests that use AI for reasoning on the correctness of LLM generated information.
• Improve testing methodologies for LLM based applications.
• Participate and designing and improving prompts used in LLM based applications.

Skills

QA Automation

Python

LLM Testing

AI Tools

Automation Frameworks

Education

Bachelor's degree in STEM fields

Job description

ICE Mortgage Technology is expanding its Quality Engineering capabilities in Atlanta to deliver a best-in-class platform for collections, loss mitigation, and homeowner support. This role emphasizes building automated test frameworks for LLM-based applications, AI-driven validation, and prompt design, with a strong focus on Python and scalable QA practices.

You will contribute to innovative testing strategies, collaborate with cross-functional teams, and stay current with emerging LLM