Automation Testing Engineer

APAR TECHNOLOGIES PTE. LTD.; Singapore; On-site; SGD 120,000 - 180,000; Full time

Job summary

APAR TECHNOLOGIES PTE. LTD. in Singapore seeks a senior QA/Data Quality Engineer to validate data using Python-based testing, Great Expectations and Pandera.

You will build automated tests for PySpark transformations, validate Delta Live Tables expectations, and ensure data integrity across the lakehouse. You will collaborate on end-to-end data reconciliation, scale testing to meet performance SLAs, and verify governance controls such as Unity Catalog RLS and data lineage.

Qualifications

• 5+ years of QA/testing experience.
• 3+ years in Big Data / Hadoop / Cloud Data Warehouse environments.
• 1+ year Databricks testing pipelines.
• Experience moving from manual SQL checks to automated Python tests.
• Automation and migration automation testing experience.

Responsibilities

• Validate data contracts with data validation frameworks.
• Write Python-based tests for PySpark transformations.
• Test Databricks workflows and job retries.
• Assess schema evolution impacts on downstream tables.
• Validate Unity Catalog RLS and column masking.

Skills

Data testing experience

Automation mindset

Python programming

Big data testing

Tools

Great Expectations

Pandera

DLT Expectations

Pytest-Spark

Unity Catalog testing

Job description

Experience Requirements

• Total QA/Testing Experience: 5+ years.
• Data Testing Experience: 3+ years specifically in Big Data, Hadoop, or CloudData Warehouse environments.
• Good to have : Databricks Experience: 1+ years of experience testing pipelines within a Databricks environment.
• Automation Focus: Proven track record of moving from manual SQL checks to automated Python-based testing frameworks.
• Migration automation testexperience using Python

Required Certifications

• Good to have: Databricks Certified Data Engineer Associate (at minimum).
• Preferred: ISTQB Foundation or Advanced Level (Test Automation Engineer).

Core Technical Skills

• 1. Data Validation Frameworks
  • Great Expectations / Pandera: Proficiency in using Python-based libraries to define data contracts and automated validation suites.
  • DLT Expectations: Deep understanding of Delta Live Tables (DLT) expectations (Fail, Drop, Quarantining bad records).
  • Advanced SQL: Expert-level SQL for complex data reconciliation, identifying duplicates, and null-value analysis across billions of records.
• 2. Python for QA (PySpark)
  • Pytest-Spark: Experience using pytest to write unit tests for PySpark transformations and logic.
  • Notebook Testing: Ability to write automated test notebooks that validate Medallion Architecture transitions (Bronze to Silver, Silver to Gold).
  • Data Reconciliation: Building Python scripts to perform source-to-target counts and checksums across distributed file systems.
• 3. Performance Integration Testing
  • Scalability Testing: Ability to validate that data pipelines meet performance SLAs when data volume spikes.
  • End-to-End Orchestration Testing: Testing the reliability of Databricks workflows and handling of job failures/retries.
  • Schema Evolution: Testing how pipelines handle upstream schema changes without breaking downstream Gold tables.
• 4. Governance Security Testing
  • Unity Catalog Validation: Testing Row-Level Security (RLS) and Column-Level Masking to ensure unauthorized users cannot see sensitive data.
  • Data Lineage: Validating that data lineage in Unity Catalog correctly reflects the movement of data across the Lakehouse.

EANumber: 11C4879

Reg. ID : R26161692