Data QA Engineer: Databricks Data Quality & Automation

Axos Clearing LLC; Manila; Hybrid; PHP 900,000 - 1,300,000; Full time

Job summary

Axos Business Center, Corp in Manila is seeking a Data QA Engineer to validate data quality across the Bronze/Silver/Gold Delta Lake layers. You will test SQL, PySpark, and Python pipelines, and contribute to the data governance and validation framework. Collaboration with data architects, BI developers, and analysts is required.

Responsibilities include building test plans, automating tests, and ensuring accuracy in Databricks notebooks and workflows while working in an Agile environment.

Qualifications

• 2+ years' experience in QA testing, preferably in a data engineering or data warehousing environment
• Data testing in SQL, PySpark/Spark SQL, and Python framework, with hands-on experience testing Databricks notebooks, Delta Lake tables, and Databricks Workflows/Jobs
• Excellent verbal and written communication skills, including ability to simplify complex concepts for technical and non-technical audience
• Superior problem-solving skills, self-motivation, and the capacity to work under pressure and on tight deadlines
• Demonstrated ability to learn, acquire and utilize new technologies, disciplines, and frameworks as needed
• Ability to build and test reliable data ETL/ELT pipelines, including Spark-based pipelines, query optimization, and dynamic SQL
• Familiarity with the medallion architecture (Bronze/Silver/Gold) and Delta Lake table validation techniques, such as schema enforcement, time travel, and ACID transactions
• Strong customer focus, excellent problem solving and analytical skills
• Ability to work independently under minimal supervision and strong track record of setting and meeting delivery commitments
• Familiarity with Agile/Scrum methodologies or working in an Agile/Scrum environment
• Databricks certification (nice to have)

Responsibilities

• Perform quality assurance and quality control duties for the Enterprise Data Management and data development life cycle
• Define, prepare, execute and implement data validation and unit and integration testing methods to ensure data quality
• Contribute to the overall test strategy and quality of the data in the Databricks Lakehouse (Enterprise Data Warehouse) and develop automated/manual test procedures
• Document bugs for developer review as well as provide supporting test results to assess the overall quality for the data product deliverable
• Work with end users, business analysts, project managers, IT, database and Tableau developers, and data architects to understand user requirements, project milestones and scope for data integration and data fulfillment initiatives to develop comprehensive test plans
• Validate technical specifications against finished data products
• Design and develop test strategies that will optimize data testing cycles and provide faster throughput without compromising on data quality
• Perform integration testing, regression testing, source to target testing and support periodic quantitative and qualitative data audits on post production data sets across Bronze, Silver, and Gold Delta Lake layers
• Define system performance standards, establish key and ancillary performance measurements, and help to ensure platform scalability
• Serve as gatekeeper for data deployments between environments and assist in automating deployments where applicable
• Validate data pipelines built with PySpark/Spark SQL and Databricks Workflows/Jobs, including notebook-based ETL/ELT processes, ensuring data accuracy across Bronze, Silver, and Gold Delta Lake layers
• Support validation of Unity Catalog access controls, data lineage, and governance policies to ensure secure, well-governed data access across the Databricks platform
• Lead specific aspects of Data Governance and Data Quality to establish data quality rules and develop data validation tests
• Leverage AI/LLM-powered tools (e.g., Claude, Copilot) to accelerate test case generation, synthetic test data creation, and code/query review, while validating AI-assisted outputs for accuracy before use in production test cycles
• Utilize Agile methodologies; participate in daily scrums, testing, code reviews, build release and other weekly meetings
• Join stakeholder meetings to communicate status, give demos, provide timelines and offer insights

Skills

QA testing

SQL testing

PySpark testing

Python scripting

Databricks

Communication

Problem solving

Bachelor's degree

Agile

Education

Bachelor's degree

Tools

Databricks

Unity Catalog

Job description

Axos Business Center, Corp in Manila is seeking a Data QA Engineer to validate data quality across the Bronze/Silver/Gold Delta Lake layers. You will test SQL, PySpark, and Python pipelines, and contribute to the data governance and validation framework. Collaboration with data architects, BI developers, and analysts is required.

Responsibilities include building test plans, automating tests, and ensuring accuracy in Databricks notebooks and workflows while working in an Agile environment.