Supply Chain Data Engineer (Databricks/PySpark/Azure)

Source Technology Limited; Dublin; On-site; EUR 73,800 - 79,335; Full time

Job summary

Source Technology Limited is seeking a Data Engineer to design, build, and maintain data pipelines feeding analytics for a large-scale supply chain visibility platform. You will own ingestion, transformation, and quality across the L0-L2 layers using Databricks/PySpark and Azure Data Factory.

You will join an Agile squad of data engineers and QA in a regulated environment prioritising data accuracy and traceability. The role supports a 2-week sprint cadence and major release cycles.

Qualifications

• Hands-on experience with Spark and Delta Lake for data processing and analysis (Databricks / PySpark).
• Proficiency in Python for SQL-based transformations and pipeline logic.
• Experience with SQL Server and/or Synapse.
• Strong data modeling fundamentals and concepts.
• Experience in performance tuning: indexing and partitioning for query performance.
• Hands-on expertise building and monitoring Azure Data Factory pipelines.
• Experience writing unit tests for code and pipelines.
• Understanding of CI/CD processes and implementation.
• Broad technical depth across databases, testing, and performance tuning.

Responsibilities

• Build & maintain data pipelines across L0-L2 using Databricks (PySpark) and Azure Data Factory.
• Develop curated data models in the L2 layer to power supply chain KPIs.
• Integrate new source systems and map raw feeds to standardized schemas.
• Implement data quality and validation controls with automated checks and traceability.
• Write and maintain unit tests; support automated test runs in CI/CD.
• Optimize Spark jobs and data workflows for performance and cost; manage Unity Catalog governance.
• Build and monitor ADF pipelines, orchestrate and troubleshoot data flows.
• Support regular releases, SIT/UAT, and hotfixes during major release cycles.
• Collaborate in agile ceremonies and document key design decisions.
• Maintain CI/CD hygiene including PRs, migrations, and quality gates.

Skills

Spark & Delta Lake

Python

SQL Server / Synapse

Data modeling

Performance tuning

Azure Data Factory

Unit testing

CI/CD

Technical depth

Tools

Databricks

Azure Data Factory

SQL Server

Synapse

neo4j

Redis

Jenkins

Bitbucket

Artifactory

Flyway

SonarQube

Jira

Confluence

Job description

Source Technology Limited is seeking a Data Engineer to design, build, and maintain data pipelines feeding analytics for a large-scale supply chain visibility platform. You will own ingestion, transformation, and quality across the L0-L2 layers using Databricks/PySpark and Azure Data Factory.

You will join an Agile squad of data engineers and QA in a regulated environment prioritising data accuracy and traceability. The role supports a 2-week sprint cadence and major release cycles.