Mid-Level QA / Testing Automation Engineer - Databricks

United States Digital Space LLC; Singapore; On-site; SGD 86,466 - 100,877; Full time

Job summary

United States Digital Space LLC is seeking a QA/Testing specialist focused on data validation for a Lakehouse environment. You will design test strategies, automate validations with Python, and collaborate with Data Engineers to diagnose Spark failures while protecting gold-layer dashboards during PySpark deployments.

Required: 5+ years QA/testing, 3+ years data testing in Big Data or cloud warehouses, and 1+ year Databricks testing. Location is Singapore; contract role with monthly pay in USD.

Qualifications

• 5+ years QA/Testing experience
• 3+ years data testing in Big Data/Hadoop/Cloud Data Warehouses
• Databricks testing experience: 1+ year
• Automation focus: shift from manual SQL checks to Python-based testing
• Migration automation testing experience using Python

Responsibilities

• Develop Test Strategy: create a comprehensive test plan for Lakehouse data integrity, accuracy and consistency
• Automate Validation: implement Python test suites in CI/CD pipelines
• Defect Analysis: document data anomalies with Data Engineers for root-cause in Spark failures
• Regression Testing: ensure new PySpark deployments do not affect Gold layer dashboards

Job description

Responsibilities

• Develop Test Strategy: Create a comprehensive test plan for the Lakehouse, focusing on Data Integrity, Accuracy, and Consistency.


• Automate Validation: Replace manual \"spot-checking\" with automated Python test suites that run as part of the CI/CD pipeline.


• Defect Analysis: Identify and document data anomalies, working closely with Data Engineers to perform root-cause analysis on Spark job failures.


• Regression Testing: Ensure that new PySpark code deployments do not impact existing Gold layer business logic or dashboard reporting.

Experience Requirements

• Total QA/Testing Experience: 5+ years. Data Testing Experience: 3+ years specifically in Big Data, Hadoop, or Cloud Data Warehouse environments.


• Must have knowledge or at least Databricks Experience: 1+ years of experience testing pipelines within a Databricks environment.


• Automation Focus: Proven track record of moving from manual SQL checks to automated Python-based testing frameworks.


• Migration automation test experience using Python

Required Certifications

• Good to have: Databricks Certified Data Engineer Associate (at minimum).


• Preferred: ISTQB Foundation or Advanced Level (Test Automation Engineer).

Core Technical Skills

Data Validation & Frameworks

• Great Expectations / Pandera: Proficiency in using Python-based libraries to define data \"contracts\" and automated validation suites.


• DLT Expectations: Deep understanding of Delta Live Tables (DLT) expectations (Fail, Drop, Quarantining bad records).


• Advanced SQL: Expert-level SQL for complex data reconciliation, identifying duplicates, and null-value analysis across billions of records.

Python for QA (PySpark)

• Pytest-Spark: Experience using pytest to write unit tests for PySpark transformations and logic.


• Notebook Testing: Ability to write automated test notebooks that validate Medallion Architecture transitions (Bronze to Silver, Silver to Gold).


• Data Reconciliation: Building Python scripts to perform \"source-to-target\" counts and checksums across distributed file systems.

Performance & Integration Testing

• Scalability Testing: Ability to validate that data pipelines meet performance SLAs when data volume spikes.


• End-to-End Orchestration Testing: Testing the reliability of Databricks Workflows and handling of job failures/retries.


• Schema Evolution: Testing how pipelines handle upstream schema changes without breaking downstream Gold tables.

Governance & Security Testing

• Unity Catalog Validation: Testing Row-Level Security (RLS) and Column-Level Masking to ensure unauthorized users cannot see sensitive data.


• Data Lineage: Validating that data lineage in Unity Catalog correctly reflects the movement of data across the Lakehouse.

Preferred Candidate Background

• Data-First Mindset: Understanding that testing a Lakehouse is about testing the data and its behavior, not just the \"UI\" or \"API\".


• Software Engineering Foundation: Candidates who know how to use Git (Branching/Merging) to manage their test code alongside the engineering team.


• Distributed Systems Knowledge: Basic understanding of Spark (shuffling, partitioning) to understand why data might be missing or duplicated in a distributed environment.

Job Type: Contract

Pay: $6,000.00 - $7,000.00 per month

Work Location: In person