Data Engineer (SC cleared)

Experis - ManpowerGroup; City Of London; Remote; GBP 60,000 - 80,000; Full time

Job summary

An innovative firm is seeking a skilled Data Engineer to join their dynamic team. This role involves working remotely on exciting projects that require a strong understanding of data concepts, data security, and complex JSON manipulation. You will leverage your expertise in Python and PySpark to develop and manage data pipelines, ensuring data integrity and compliance with GDPR. Join a collaborative Agile environment where your analytical skills and problem-solving abilities will be highly valued. If you are passionate about data engineering and eager to make an impact, this opportunity is perfect for you.

Qualifications

• Strong understanding of data concepts and data types.
• Experience with Data Pipelines using Python/PySpark frameworks.
• Knowledge of Data Security principles and GDPR.

Responsibilities

• Develop and manage data pipelines ensuring data integrity and security.
• Utilize Python and PySpark for data processing tasks.
• Collaborate in an Agile environment to deliver data solutions.

Skills

Data Concepts

JSON Manipulation

Python/PySpark

Data Security Principles

Analytical Skills

Linux System Administration

Agile Methodologies

Problem Solving

Tools

Azure Databricks

Apache Spark

PowerBI

JIRA

Git

Job description

Data Engineer
4 months - with extensions
Remote
Active SC clearance required
£640 per day inside ir35

REQUIRED

1. Strong understanding of data concepts - data types, data structures, schemas (both JSON and Spark), schema management etc.
2. Strong understanding of complex JSON manipulation.
3. Experience working with Data Pipelines using a custom Python/PySpark frameworks.
4. Strong understanding of the 4 core Data categories (Reference, Master, Transactional, Freeform) and the implications of each, particularly managing/handling Reference Data.
5. Strong understanding of Data Security principles - data owners, access controls - row and column level, GDPR etc including experience of handling sensitive datasets.
6. Strong problem solving and analytical skills, particularly able to demonstrate these intuitively (able to work a problem out, not follow a work instruction to resolve).
7. Experience working in a support role would be beneficial, particularly able to demonstrate incident triage and handling skills/knowledge (SLAs etc).
8. Fundamental linux system administration knowledge - ssh keys and config etc, Bash CLI and scripting, Environment variables.
9. Experience using browser based IDEs (Jupyter Notebooks, RStudio etc).
10. Experience working in a dynamic Agile environment (SAFE, scrum, sprints, JIRA etc).
11. Python (as a programming language, not just being able to write basic scripts).

LANGUAGES / FRAMEWORKS

1. JSON
2. YAML
3. Python (as a programming language, not just able to write basic scripts)
4. Pydantic experience DESIRABLE
5. SQL
6. PySpark
7. Delta Lake
8. Bash (both CLI usage and scripting)
9. Git
10. Markdown
11. Scala DESIRABLE
12. Azure SQL Server as a HIVE Metastore DESIRABLE

TECHNOLOGIES

1. Azure Databricks
2. Apache Spark
3. Delta Tables
4. Data processing with Python
5. PowerBI (Integration / Data Ingestion)
6. JIRA

If you meet the above requirements, please apply for the vacancy to be contacted by an Experis Consultant. If you haven't been contacted within 2 weeks of application, please consider the vacancy closed.