Data Engineer for Shelf Analytics

Luxoft; Remote; GBP 50,000 - 70,000; Full time

Job summary

A technology consulting firm in the United Kingdom is seeking an experienced Data Engineer to join the Shelf Analytics project. The role involves designing, developing, and maintaining scalable data pipelines and workflows using Databricks and PySpark. Candidates should have strong programming skills in Python, hands-on experience with Databricks, and solid knowledge of SQL. This position aims to enhance product visibility through effective data management and analytics, primarily in a retail setting.

Qualifications

• Strong programming skills in Python and PySpark.
• Hands-on experience with Databricks workflow and cluster management.
• Solid knowledge of SQL with experience in Spark SQL.

Responsibilities

• Design and maintain data pipelines and workflows using Databricks.
• Implement object-oriented Python solutions.
• Develop and optimize complex SQL queries.

Skills

Python programming

PySpark

Databricks

SQL

Spark SQL

Azure Storage

GitHub

Unit testing

Job description

Project description

We are looking for an experienced Data Engineer to join the Shelf Analytics project – a data-driven application designed to analyze how P&G products are positioned on store shelves. The primary objective of the solution is to improve product visibility, optimize in-store execution, and ultimately increase sales by combining shelf layout data with sales insights. As a Data Engineer, you will play a key role in building, maintaining, and enhancing scalable data pipelines and analytics workflows that power shelf-level insights. You will work closely with analytics and business stakeholders to ensure high-quality, reliable, and performant data solutions.

Responsibilities

• Design, develop, and maintain data pipelines and workflows using Databricks and PySpark
• Read, understand, and extend existing codebases; independently develop new components for Databricks workflows
• Implement object-oriented Python solutions (classes, inheritance, reusable modules)
• Develop and maintain unit tests to ensure code quality and reliability
• Work with Spark SQL and SQL Server Management Studio to create and optimize complex queries
• Create and manage Databricks workflows, clusters, databases, and tables
• Handle data storage and access management in Azure Data Lake Storage (ADLS), including ACL permissions
• Collaborate using GitHub, following CI/CD best practices and working with GitHub Actions
• Support continuous improvement of data engineering standards, performance, and scalability

Skills – must have

• Strong programming skills in Python and PySpark
• Hands-on experience with Databricks (workflows, clusters, tables, databases)
• Solid knowledge of SQL and experience with Spark SQL and SQL Server Management Studio
• Experience with pandas, dbx, and unit testing frameworks
• Practical experience working with Azure Storage (ADLS) and access control (ACLs)
• Proficiency with GitHub, including CI/CD pipelines and GitHub Actions
• Ability to work independently, analyze existing solutions, and propose improvements

Nice to have

Experience with retail, CPG, or shelf analytics–related solutions

Familiarity with large-scale data processing and analytics platforms

Strong communication skills and a proactive, problem-solving mindset