Senior Data Engineer - Databricks & ETL Pipelines (Hybrid)

UST; Leeds; Hybrid; GBP 70,000 - 110,000; Full time

Job summary

UST is seeking an experienced Data Engineer to design, build and optimise enterprise-scale data ingestion pipelines. You will work with Databricks, PySpark and SparkSQL to transform data from vendor platforms into the client data ecosystem, ensuring secure, governed, and accessible data for reporting and analytics.

The role requires strong experience with SFTP-based ingestion, cloud technologies and data modelling, and collaboration with analysts, product owners and architects in an Agile

Qualifications

• Proven experience designing, building and supporting enterprise-scale data ingestion pipelines and ETL/ELT solutions.
• Strong hands-on experience with Databricks, PySpark and SparkSQL.
• Experience developing and supporting secure data integrations using SFTP and other file-based or API-driven ingestion mechanisms.
• Experience ingesting and processing structured, semi-structured and unstructured data from internal and third-party source systems.
• Strong understanding of data modelling, transformation techniques and data warehousing principles.
• Experience working with cloud-based data lake and analytics platforms.
• Strong understanding of batch and near real-time data processing patterns.
• Experience conducting data profiling, discovery and validation activities to assess data quality, completeness and suitability for business requirements.
• Experience implementing data quality checks, reconciliations and monitoring processes.
• Ability to investigate and resolve ingestion, transformation and data quality issues identified during testing, UAT or production support.
• Understanding of data governance, security, data lineage and documentation standards.
• Experience producing technical documentation and operational handover materials.
• Strong stakeholder engagement skills with the ability to work effectively across business, architecture, engineering and analytics teams.
• Experience working within Agile delivery environments.
• Knowledge of source control, CI/CD practices and release management processes.
• Ability to work independently while collaborating effectively within cross-functional squads.
• Experience integrating data from retail technology platforms, IoT devices or third-party vendor systems.
• Experience working with AI Camera, Computer Vision or Electronic Shelf Edge Label (eSEL) technologies.
• Knowledge of Azure Data Lake, Azure Data Factory and related Azure data services.
• Experience supporting reporting, analytics or BI solutions through the creation of trusted and governed data assets.
• Experience working within large-scale retail or data transformation programmes.

Responsibilities

• Design, build and optimise robust data ingestion pipelines to acquire, transform and load data vendor platforms into client's data ecosystem.
• Develop scalable data engineering solutions using Databricks, PySpark, SparkSQL and associated cloud technologies.
• Build and maintain secure, reliable and automated data ingestion processes from external vendor systems, including SFTP-based file transfers and other integration methods.
• Ensure data is landed, structured, governed and accessible to support reporting, analytics and business use cases.
• Work with Business Analysts, Product Owners, Architects and delivery squads to translate business requirements into technical data solutions.
• Support data discovery, profiling and validation activities to understand source data structures, data quality issues and data gaps.
• Develop and maintain data transformations, curated datasets and data models required to support reporting and analytical use cases.
• Monitor, troubleshoot and resolve data ingestion issues, defects and enhancements identified during development, testing, UAT and production support.
• Ensure solutions comply with data architecture standards, engineering best practices, security requirements and governance frameworks.
• Produce clear technical documentation for data ingestion processes, data flows and operational support requirements.
• Provide comprehensive handover documentation and knowledge transfer to the Data Support team following delivery of data ingestion pipelines.
• Collaborate within Agile delivery teams, actively contributing to sprint planning, stand-ups, retrospectives and continuous improvement activities.
• Identify opportunities to improve pipeline performance, automation, scalability and maintainability through process and technology enhancements.
• Support knowledge sharing and contribute to Engineering and Data Communities of Practice.

Skills

Databricks

PySpark

SparkSQL

ETL/ELT pipelines

SFTP

Azure Data Lake

Azure Data Factory

Data modelling

Data governance

CI/CD

Agile

Cloud platforms

Data profiling

Tools

Azure Data Lake

Azure Data Factory

Git / CI/CD

Job description

UST is seeking an experienced Data Engineer to design, build and optimise enterprise-scale data ingestion pipelines. You will work with Databricks, PySpark and SparkSQL to transform data from vendor platforms into the client data ecosystem, ensuring secure, governed, and accessible data for reporting and analytics.

The role requires strong experience with SFTP-based ingestion, cloud technologies and data modelling, and collaboration with analysts, product owners and architects in an Agile