QA Automation Engineer - AI & LLM Testing

ICE; Jacksonville (FL); On-site; USD 85,000 - 110,000; Full time

Job summary

ICE Mortgage Technology is building a world-class Quality Engineering team to elevate our Collections and Loss Mitigation offerings and to empower loan servicers with AI-driven testing capabilities. The role focuses on automating tests for LLM-based applications, developing robust frameworks, and ensuring quality across the platform.

The ideal candidate has a STEM degree and 3+ years in QA automation, strong Python skills, and experience using AI tools to build test automation.

Qualifications

• Bachelor’s degree in a STEM field.
• 3+ years in QA and QA automation.
• Proficiency in Python.
• Ability to learn quickly and adapt to new technologies, with focus on LLM tech.
• Excellent communication and teamwork skills.

Responsibilities

• Implement automated test frameworks for LLM-based applications.
• Develop automated tests using AI for reasoning on correctness of LLM outputs.
• Improve testing methodologies for LLM-based applications.
• Contribute to designing and improving prompts for LLM-based applications.

Skills

QA automation

Python

LLM testing

AI tools

Automation frameworks

Education

Bachelor’s degree in STEM

Tools

Agentic AI tools

Job description

ICE Mortgage Technology is building a world-class Quality Engineering team to elevate our Collections and Loss Mitigation offerings and to empower loan servicers with AI-driven testing capabilities. The role focuses on automating tests for LLM-based applications, developing robust frameworks, and ensuring quality across the platform.

The ideal candidate has a STEM degree and 3+ years in QA automation, strong Python skills, and experience using AI tools to build test automation.