DATA ENGINEER - SC CLEARED

LA International; London; Remote; GBP 40,000 - 80,000; Full time

Job summary

Join a forward-thinking consultancy as a Data Engineer, where your expertise in data concepts and Python will be pivotal in developing and managing data pipelines. This fully remote role with occasional travel to London offers an exciting opportunity to work with cutting-edge technologies like Azure Databricks and Apache Spark. You'll be part of a dynamic team, contributing to high-impact projects while ensuring data security and compliance. If you're passionate about data and thrive in an Agile environment, this role is perfect for you. Embrace the chance to make a significant impact in a company recognized for its excellence in IT recruitment and project solutions.

Qualifications

• Strong understanding of data types, structures, and schemas.
• Experience with complex JSON manipulation and data pipelines.

Responsibilities

• Develop and manage data pipelines using Python and PySpark.
• Ensure data security and compliance with GDPR regulations.

Skills

Data concepts understanding

JSON manipulation

Python/PySpark frameworks

Data Security principles

Problem solving skills

Linux system administration

Agile environment experience

Tools

Azure Databricks

Apache Spark

PowerBI

JIRA

Git

Job description

REQUIRED: DATA ENGINEER - SC CLEARED
LOCATION: FULLY REMOTE WITH OCCASIONAL TRAVEL TO CENTRAL LONDON
IR35 STATUS: INSIDE
DURATION: 6 MONTH INITIAL CONTRACT

Seeking a Data Engineer who has a strong understanding of data concepts:

1. Data types, data structures, schemas (both JSON and Spark), schema management
2. Strong understanding of complex JSON manipulation
3. Experience working with Data Pipelines using custom Python/PySpark frameworks
4. Strong understanding of the 4 core Data categories (Reference, Master, Transactional, Freeform) and the implications of each, particularly managing/handling Reference Data
5. Strong understanding of Data Security principles - data owners, access controls - row and column level, GDPR including experience of handling sensitive datasets
6. Strong problem solving and analytical skills, particularly able to demonstrate these intuitively
7. Experience working in a support role would be beneficial, particularly able to demonstrate incident triage and handling skills/knowledge (SLAs etc)
8. Fundamental Linux system administration knowledge - ssh keys and config, Bash CLI and scripting, Environment variables
9. Experience using browser-based IDEs (Jupyter Notebooks, RStudio)
10. Experience working in a dynamic Agile environment (SAFE, scrum, sprints, JIRA)

Languages / Frameworks:

1. JSON
2. YAML
3. Python (as a programming language, not just able to write basic scripts; Pydantic experience would be a bonus)
4. SQL
5. PySpark
6. Delta Lake
7. Bash (both CLI usage and scripting)
8. Git
9. Markdown
10. Scala (bonus, not compulsory)
11. Azure SQL Server as a HIVE Metastore (bonus)

Technologies:

1. Azure Databricks
2. Apache Spark
3. Delta Tables
4. Data processing with Python
5. PowerBI (Integration / Data Ingestion)
6. JIRA

Due to the nature and urgency of this post, candidates holding or who have held high-level security clearance in the past are most welcome to apply. Please note successful applicants will be required to be security cleared prior to appointment which can take up to a minimum of 10 weeks.

LA International is a HMG approved ICT Recruitment and Project Solutions Consultancy, operating globally from the largest single site in the UK as an IT Consultancy or as an Employment Business & Agency depending upon the precise nature of the work. LA International welcomes applications from all sections of the community and from people with diverse experience and backgrounds.

Award Winning LA International, winner of the Recruiter Awards for Excellence, Best IT Recruitment Company, Best Public Sector Recruitment Company, and overall Gold Award winner, has now secured the most prestigious business award that any business can receive, The Queens Award for Enterprise: International Trade, for the second consecutive period.