Data Engineer

Axiom Software Solutions Limited; United Kingdom; Remote; GBP 45,000 - 65,000; Full time

Job summary

A leading software solutions provider in the UK is looking for a Data Engineer for a 6-month remote contract inside IR35. The ideal candidate will have a robust understanding of data structures, strong skills in Python and PySpark for Data Pipeline creation, and experience with GDPR compliance. This role requires expertise in JSON manipulation and data security principles.

Qualifications

• Strong understanding of data concepts and structures.
• Experience managing sensitive datasets and GDPR compliance.
• Proven analytical skills with problem-solving abilities.

Skills

Data Engineering understanding

JSON manipulation

Data Pipelines with Python/PySpark

Data Security principles

Problem solving skills

Tools

Azure Databricks

Apache Spark

Delta Tables

PowerBI

Git

Job description

Overview

Job title: Data Engineer

Location: UK

Duration: 6-month contract Inside IR35

Working Mode: Fully Remote

Vetting: Active SC Clearance

Job Description:

Data Engineer Skills and Technologies

• Data Engineering: Strong understanding of data concepts, data types, data structures, schemas (both JSON and Spark), schema management
• Strong understanding of complex JSON manipulation
• Experience working with Data Pipelines using a custom Python/PySpark frameworks
• Strong understanding of the 4 core Data categories (Reference, Master, Transactional, Freeform) and the implications of each, particularly managing/handling Reference Data
• Strong understanding of Data Security principles, data owners, access controls
• Row and column level, GDPR compliance including experience of handling sensitive datasets
• Strong problem solving and analytical skills, particularly able to demonstrate these intuitively

Required Skills:

• Languages / Frameworks
  • JSON
  • YAML
  • Python (as a programming language, not just able to write basic scripts. Pydantic experience would be a bonus.)
  • SQL
  • PySpark
  • Delta Lake
  • Bash (both CLI usage and scripting)
  • Git
  • Markdown
  • Scala (bonus, not compulsory)
  • Azure SQL Server as a HIVE Metastore (bonus)
• Technologies
  • Azure Databricks
  • Apache Spark
  • Delta Tables
  • Data processing with Python
  • PowerBI (Integration / Data Ingestion)
  • JIRA