Senior Software QA Engineer

Crimson Education; Auckland; Hybrid; NZD 90,000 - 130,000; Full time

Benefits offered by this job

Flexible working environment

Internal promotions and role changes

$1000 training budget per year

Psychologist on staff

Fireside chats and workshops

Radical Candor culture

Global company with 28 markets

Job summary

Crimson Education, a global college admissions leader, seeks a Software QA Automation engineer to design and scale AI/ML testing across UI, API, and integration layers. You will embed automated tests in CI/CD, define coverage, and push for high-quality deployments in a fast-growing environment.

You should have 4+ years in QA Automation with AI/LLM experience, strong Python or JavaScript/TypeScript skills, and hands-on tools like Playwright, Cypress, Selenium, Postman, RestAssured, PyTest.

Qualifications

• 4+ years of experience in Software QA Automation, with at least 1–2 years focused on testing AI/ML-powered applications or LLM integrations.

Responsibilities

• Design, build, and scale end-to-end automation test frameworks (UI, API, and integration).
• CI/CD integration: bake automated tests into pipelines for rapid deployments.
• Define, track, and scale test coverage metrics for high-purity API and critical UI paths.
• Conduct API performance, load, and stress testing to maintain scalable infrastructure.
• Design testing strategies for non-deterministic AI systems and evaluate outputs for accuracy, bias, safety and hallucinations.
• Establish prompt regression benchmarks to prevent feature breaks.
• Collaborate with Data Scientists/ML Engineers to monitor model performance and drift.
• Hunt edge cases and simulate adversarial inputs to ensure security.

Skills

QA Automation

Python

JavaScript/TypeScript

AI/ML Testing

API Testing

Ragas/TruLens/Deepchecks

Vector Databases

Bachelor’s Degree

Education

Bachelor’s degree in CS/Engineering

Tools

Playwright

Cypress

Selenium

Postman

RestAssured

PyTest

Job description

Jora New Zealand will close on 9th September 2026. Thank you for being with us, we are cheering you on as you continue your career journey.

Crimson Education – Auckland, North Island

Crimson Education is the world’s leading college admissions consulting firm, with over 1,490 Ivy League offers and 2,410 to the US Top 15. We help ambitious students gain admission to the world’s top universities through expert-led guidance and proven, data-driven strategies. Crimson students are 7x more likely to get into the Ivy League than their peers. We were recently featured on the front page of the Wall Street Journal.

Crimson is the only college admissions consultancy that brings together:

• Former Ivy League and Top 20 admissions officers to rigorously review and refine applications

• Professors and PhD teaching fellows from leading universities to guide students through original, independent research — with pathways to selective peer-reviewed publication or conference presentation

• Past ISEF winners and judges who coach students to compete for state, national, and international science fair awards

• Capstone project mentors who help students design and scale leadership initiatives with real-world impact, measurable outcomes, and credible external validation

We’re backed by leading VC firms, including Tiger Global, Heal Partners, IceHouse Ventures, and Movac, and recently closed a USD $40M Series D funding round at a USD $640M valuation. We now operate across 21 markets worldwide, including the US, Canada, UK, Singapore, Japan, Hong Kong, Australia, and New Zealand.

The Role

• Design & Build: Develop, maintain, and scale robust, end-to-end automation test frameworks from scratch (UI, API, and Integration layers).

• CI/CD Integration: Seamlessly bake automated test suites into our CI/CD pipelines to ensure rapid, high-confidence deployments.

• Coverage Strategy: Define, track, and scale test coverage metrics (targeting high-purity API and critical UI path coverage) to ensure regression-free deployments.

• Performance & Scale: Conduct API performance, load, and stress testing to ensure our infrastructure keeps up with demand.

• AI/LLM Validation: Design testing strategies for non-deterministic systems. You’ll evaluate LLM outputs for accuracy, bias, safety, and hallucination rates.

• Prompt Regression Testing: Establish benchmarks to ensure that tweaking an AI prompt or upgrading a foundational model doesn't secretly break existing features.

• Data & Model Drift Monitoring: Collaborate with Data Scientists and ML Engineers to monitor model performance metrics in production.

• Edge Case Hunting: Actively simulate adversarial inputs (prompt injections, jailbreaks) to ensure our AI features remain secure and guardrailed.

What skills and experience are required?

• 4+ years of experience in Software QA Automation, with at least 1–2 years specifically focused on testing AI/ML-powered applications or LLM integrations.

• Strong programming proficiency in Python (highly preferred for AI workflows) or JavaScript/TypeScript.

• Hands-on experience with modern automation tools (e.g., Playwright, Cypress, Selenium).

  • Expertise in API testing tools (e.g., Postman, RestAssured, PyTest).

• Familiarity with AI evaluation frameworks (e.g., Ragas, TruLens, Deepchecks) or a proven ability to write custom evaluation scripts for probabilistic outputs.

• Basic understanding of vector databases, embeddings, and how data pipelines feed into ML models.

• Bachelor’s degree in Computer Science, Engineering, or equivalent practical experience.

Bonus Points

• Familiarity with cloud platforms (AWS, GCP, or Azure) and containerization (Docker, Kubernetes).

• A healthy obsession with AI safety and ethics.

Why work for Crimson?

• Rapidly growing start up, with a flexible working environment where you will be empowered to structure how you work

• Limitless development and exposure - our internal promotions/role changes made 33% of all recruitment last year

• $1000 individual training budget per year, we love to ‘Level Up’ (it’s one of our core values)!

• Psychologist on staff

• Insightful fireside chats and workshops to help support our high-performing and ambitious team

• Radical Candor is a feedback approach we live by

• We’re a global player with 28 markets (and growing) across the globe. Most roles have the option to work from one of our many offices or remotely!

If you're passionate about growing in a fast-paced, collaborative environment and want to work with cutting-edge technology, then we'd love to hear from you!