Automation Test Lead

JR United Kingdom; Reading; On-site; GBP 40,000 - 80,000; Full time

Job summary

An established industry player is seeking a meticulous QA Engineer to enhance the quality of data-driven systems. In this dynamic role, you'll collaborate with cross-functional teams, ensuring compliance with data privacy standards while actively participating in the Agile development lifecycle. Your expertise in test automation frameworks and data validation techniques will be crucial as you design comprehensive test plans and execute automated tests for complex ETL workflows. Join a forward-thinking company where your contributions will directly impact the reliability and performance of innovative data solutions.

Qualifications

• Experience in testing Databricks, Apache Spark, and Delta Lake.
• Strong knowledge of Python, SQL, and Scala for QA automation.

Responsibilities

• Design and execute test plans for functional and performance testing.
• Validate data quality and compliance with data privacy regulations.

Skills

Databricks

Apache Spark

Python

SQL

Scala

Test Automation

Data Validation

Agile Methodologies

API Testing

Cloud Platforms

Tools

GitLab

PyTest

Robot Framework

Postman

Docker

JIRA

Job description

Social network you want to login/join with:

We are seeking a detail-oriented and proactive QA Engineer to join our team and collaborate closely with the Solution Architect, Lead Data Engineer, and cross-functional team members to ensure the quality and performance of our data-driven systems. The QA Engineer will actively contribute throughout the Agile development lifecycle, participating in planning, refinement, and review ceremonies. This role involves validating system functionalities, ensuring compliance with data privacy standards, and supporting automated and manual testing processes. The ideal candidate has a passion for quality assurance within fast-paced, iterative Agile environments.

Key Responsibilities:

• Test Planning & Execution - Design, write, and execute comprehensive test plans, test cases, and test scripts for functional, integration, regression, and performance testing.
• Test Automation & Scripting
• Build and maintain automated test suites to support regression testing and CI/CD pipelines. Use Python-based frameworks (e.g., PyTest) and integrate with GitLab.

Design and implement automated testing frameworks for Databricks pipelines and ETL workflows.

• Validate Hive Metastore and Unity Catalog configurations to ensure data consistency and security.
• Perform data quality assurance, including completeness, accuracy, and transformation validation.
• Develop and execute unit, integration, and end-to-end tests for Apache Spark processing.
• Test stateful processing mechanisms and structured streaming pipelines for scalability.
• Automate pipeline monitoring, failure detection, and alerting mechanisms.
• Validate pause/resume mechanisms based on responses from external API calls.
• Ensure compliance with enterprise data governance and security policies.
• Collaborate closely with Data Engineers and DevOps teams to enhance reliability and performance.
• Conduct root cause analysis and post-deployment validation for production pipelines.
• System Review & Collaboration - Engage with the Solution Architect and Lead Data Engineer to understand system architecture, workflows, and business requirements. Review stories and acceptance criteria during grooming sessions.
• Validation of New Features - Test new feature implementations in each sprint cycle and provide timely feedback. Ensure features meet acceptance criteria and function correctly within the broader system.
• Data Privacy & Compliance Testing - Validate system changes for compliance with data privacy regulations (e.g., GDPR) and verify correct implementation of data masking, encryption, and access controls.
• Defect Management - Identify, log, track, and manage defects using issue-tracking tools. Collaborate with developers to troubleshoot and resolve issues quickly within sprint timelines.
• Documentation & Impact Analysis - Maintain detailed records of test cases, execution results, defects, and QA sign-off. Provide impact assessments for production changes and test results.
• Version Control & Collaboration - Work within GitLab repositories to manage test code and documentation, adhering to Agile workflows and code management best practices.
• Agile Participation - Actively contribute in Agile ceremonies including daily stand-ups, sprint planning, backlog refinement, and sprint reviews. Collaborate with developers, product owners, and business analysts to understand user stories and define test criteria.

Required Skills and Experience:

• Experience in Databricks, Apache Spark, and Delta Lake testing strategies.
• Strong knowledge of Python, SQL, and Scala for QA automation.
• Familiarity with Hive Metastore, Unity Catalog, and data governance best practices.
• Hands-on experience in test automation frameworks, such as PyTest, Robot Framework, or custom-built solutions.
• Proficiency in data validation techniques, including schema checks, transformation validation, and anomaly detection.
• Understanding of event-driven architectures and stateful processing in streaming applications.
• Experience with API testing for external integrations within Databricks pipelines.
• Knowledge of cloud platforms like Azure, AWS, or Google Cloud in data engineering environments.
• Experience writing automated and manual test cases.
• Strong scripting skills in Python for automation and data validation.
• Experience working with Databricks Notebooks and validating data transformations.
• Familiarity with AWS S3, including validating file ingestion, metadata, and access control.
• Understanding of data privacy principles and experience in validating secure data handling.
• Experience using GitLab for version control and collaboration.
• Solid understanding of Agile methodologies and experience working in Scrum or Kanban environments.

Preferred (Nice to Have):

• Experience with CI/CD pipelines for Databricks-based workflows.
• Prior work in data-intensive projects, ensuring performance, reliability, and scalability.
• Strong analytical skills to diagnose and improve data quality and testing efficiency.
• Experience testing RESTful APIs using tools like Postman or REST Assured.
• Exposure to Docker containers and Ansible for QA environments and deployment validations.
• Familiarity with Agile test management tools like JIRA.

Soft Skills:

• Strong analytical thinking with attention to detail in identifying edge cases and critical bugs.
• Clear and concise communication skills for reporting issues and discussing technical requirements.
• Collaborative, adaptable, and a team player who thrives in a fast-paced Agile environment.
• Capable of working independently and proactively managing tasks within the sprint.