QA Data Engineering Lead — Databricks & ETL Leader (Hybrid)

SMBC Group; Tralee; Hybrid; EUR 90,000 - 150,000; Full time

Benefits offered by this job

Hybrid work model

Accommodations during candidacy

Job summary

SMBC Group in Tralee, Ireland is seeking an experienced QA Data Engineering Lead to ensure data quality on Databricks platforms. This hands-on technical leadership role focuses on validating complex data pipelines and large-scale datasets using SQL, Python, PySpark and Databricks.

The role leads the overall data QA strategy and testing governance, collaborating with global technology and business teams to ensure release readiness and high data quality.

Qualifications

• Bachelor’s degree in Computer Science, Engineering, Data or a related discipline, or equivalent relevant experience.
• 7+ years of experience across database, ETL, data warehouse or data-quality testing, with demonstrated experience leading complex QA delivery.
• Proven experience leading QA teams and providing technical direction across complex data environments.
• Strong hands-on experience with Databricks, advanced SQL and Python, including PySpark and Pandas.
• Strong SQL capability including window functions, complex joins and CTEs for large-scale data validation.
• Strong experience with Databricks Notebooks and cluster management.
• Strong understanding of ETL processes, data pipelines, data warehouse concepts and end-to-end data validation.
• Experience validating data across Databricks Medallion Architecture, including Bronze, Silver and Gold layers.
• Experience working within Azure or AWS environments.
• Experience developing test strategies, establishing QA governance and assessing release readiness.
• Experience with Jira and working within Agile/Scrum delivery environments.
• Proven ability to introduce and improve data-testing processes, standards and automation practices.
• Strong analytical and problem-solving skills with the ability to operate across complex data environments.
• Strong communication and stakeholder management skills with experience working across global teams and engaging senior stakeholders.

Responsibilities

• Lead overall data QA strategy and testing governance.
• Manage and mentor a team of Database QA Engineers.
• Develop testing plans for complex data pipelines and ETL processes.
• Validate data across Bronze, Silver and Gold Databricks layers.
• Perform end-to-end data validation and reconciliation across stages and systems.
• Write advanced SQL queries for data validation.
• Develop Python scripts using PySpark and Pandas for data validation.
• Review Databricks jobs and pipeline runs for quality and reliability.
• Investigate pipeline failures and data inconsistencies with engineering teams.
• Define, document and track data-testing scenarios and test cases.
• Identify, document and manage defects through Jira.
• Collaborate with Data Engineers and developers to resolve issues.
• Provide oversight of significant data-quality issues and escalate as needed.

Skills

QA leadership

Team leadership

Stakeholder management

Analytical thinking

Problem solving

Communication

Education

Bachelor’s degree in CS/Engineering/Data

Tools

Databricks

SQL

Python

PySpark

Pandas

Databricks Notebooks

Azure

AWS

Jira

Job description

SMBC Group in Tralee, Ireland is seeking an experienced QA Data Engineering Lead to ensure data quality on Databricks platforms. This hands-on technical leadership role focuses on validating complex data pipelines and large-scale datasets using SQL, Python, PySpark and Databricks.

The role leads the overall data QA strategy and testing governance, collaborating with global technology and business teams to ensure release readiness and high data quality.