Automation Tester

TEKsystems; Burgess Hill; On-site; GBP 45,000 - 60,000; Part time

Job summary

TEKsystems in Burgess Hill, UK is seeking an Automation Tester to design and implement testing strategies for AI/LLM-based systems. You will develop evaluation frameworks for AI outputs and assess accuracy, relevance, and edge cases, while ensuring consistency and reliability.

Collaborating closely with engineering, product, and data teams, you will help define quality standards for non-deterministic AI systems and help drive automated evaluation techniques in a contract setting.

Qualifications

• Design and implement testing strategies for AI/LLM-based systems.
• Develop and apply evaluation frameworks for AI outputs.
• Assess and validate: accuracy and relevance of responses; consistency and reliability; hallucinations and edge cases.
• Work with LLM-as-judge patterns and automated evaluation techniques.
• Define quality standards for non-deterministic AI systems.
• Collaborate closely with engineering, product, and data teams.

Responsibilities

• Design and implement testing strategies for AI/LLM-based systems.
• Develop and apply evaluation frameworks for AI outputs.
• Assess and validate: accuracy and relevance of responses; consistency and reliability; hallucinations and edge cases.
• Work with LLM-as-judge patterns and automated evaluation techniques.
• Define quality standards for non-deterministic AI systems.
• Collaborate with engineering, product, and data teams.

Skills

Testing strategies

Evaluation frameworks

LLM-as-judge patterns

Quality standards

Automation

QA collaboration

Tools

Java

Kotlin

Kafka

Job description

Description

• Design and implement testing strategies for AI/LLM-based systems
• Develop and apply evaluation frameworks for AI outputs
• Assess and validate:
  • Accuracy and relevance of responses
  • Consistency and reliability
  • Hallucinations and edge cases
• Work with LLM-as-judge patterns and automated evaluation techniques
• Define quality standards for non-deterministic AI systems
• Collaborate closely with engineering, product, and data teams

QE – Automation (API & Event-Based)

Key Responsibilities

• Design and implement testing strategies for AI/LLM-based systems
• Develop and apply evaluation frameworks for AI outputs
• Assess and validate:
  • Accuracy and relevance of responses
  • Consistency and reliability
  • Hallucinations and edge cases
• Work with LLM-as-judge patterns and automated evaluation techniques
• Define quality standards for non-deterministic AI systems
• Collaborate closely with engineering, product, and data teams

Required Skills & Experience

• Design and implement testing strategies for AI/LLM-based systems
• Develop and apply evaluation frameworks for AI outputs
• Assess and validate:
  • Accuracy and relevance of responses
  • Consistency and reliability
  • Hallucinations and edge cases
• Work with LLM-as-judge patterns and automated evaluation techniques
• Define quality standards for non-deterministic AI systems
• Collaborate closely with engineering, product, and data teams

Additional Details

Industry Background: Open (no specific industry Experience required)

3x days a week in Burgess hill office.

Skills

• java
• kotlin
• kafka
• Testing
• Automation
• Quality assurance

Job Title:

Automation Tester

Location:

Burgess Hill, UK

Job Type:

Contract