QA Automation Engineer – LLM & AI Testing

ICE; Dallas (TX); On-site; USD 110,000 - 140,000; Full time

Job summary

ICE Mortgage Technology is building a world-class Quality Engineering organization to elevate our Collections and Loss Mitigation offerings, and to empower loan servicers with tools, automation, and insights to identify, prioritize, and focus on homeowners needing support.

We seek a QA automation professional with 3+ years of experience and Python proficiency to implement automated test frameworks for LLM-based applications and to contribute to AI-driven testing strategies.

Qualifications

• Bachelor’s degree in STEM fields.
• 3+ years in QA and QA automation.
• Experience using Agentic AI tools to build test automation tools.
• Proficiency in Python.
• Ability to learn quickly and adapt to new technologies, with focus on LLM related developments.
• Ability to independently learn new complex business domains.
• Strong problem-solving and attention to detail.
• Excellent communication and teamwork skills.
• Ability to communicate fluently and professionally in complex technical and business contexts.

Responsibilities

• Implement automated test frameworks and own existing frameworks for LLM-based applications.
• Implement automated tests that use AI for reasoning on the correctness of LLM generated information.
• Improve testing methodologies for LLM based applications.
• Participate and design and improve prompts used in LLM based applications.

Skills

QA automation

Python

Agentic AI tools

Communication

Problem solving

Learning agility

Teamwork

LLM testing

Education

Bachelor's degree in STEM fields

Tools

LLM testing frameworks

Automation tools

CI/CD

Job description

ICE Mortgage Technology is building a world-class Quality Engineering organization to elevate our Collections and Loss Mitigation offerings, and to empower loan servicers with tools, automation, and insights to identify, prioritize, and focus on homeowners needing support.

We seek a QA automation professional with 3+ years of experience and Python proficiency to implement automated test frameworks for LLM-based applications and to contribute to AI-driven testing strategies.