UK Data Engineer - DV clearance required

Atreides; Crewe; Hybrid; GBP 60,000 - 80,000; Full time

Job summary

A dynamic technology company in the UK is seeking an experienced AWS or Azure Data Engineer. The role focuses on designing and maintaining scalable cloud infrastructure while optimizing data pipelines. Candidates should have 3+ years of relevant experience and strong skills in Infrastructure as Code, data engineering, and data orchestration tools. Competitive salary and comprehensive benefits are offered, along with a flexible hybrid work environment.

Benefits

Competitive salary

Comprehensive health, dental, and vision insurance

Flexible hybrid work environment

Flexible hours

Work travel opportunities

Competitive vacation time

Parental leave

Qualifications

• 3+ years of experience in AWS/Azure Infrastructure and/or SRE roles.
• Strong background in Infrastructure as Code (IaC).
• Experience with data orchestration tools such as Dagster, Airflow, or Prefect.
• Familiarity with geospatial data formats like Parquet/GeoParquet.
• Excellent problem-solving skills.

Responsibilities

• Design and maintain AWS or Azure infrastructure with a focus on IaC.
• Develop and maintain scalable data pipelines using Spark, Scala, and Python.
• Monitor and troubleshoot data pipelines and infrastructure.

Skills

Infrastructure as Code (IaC)

Data engineering

Spark

Scala

Python

AWS

Azure

Problem-solving

Communication skills

Tools

Pulumi

Terraform

Docker

Git

Jenkins

Job description

Overview

Job Title: AWS or AZURE Data Engineer

Onsite: once a month to West Midlands area UK

Required Active Clearance: Yes

Company Overview: Atreides helps organizations transform large and complex multi-modal datasets into information-rich geo-spatial data subscriptions that can be used across a wide spectrum of use cases. Currently, Atreides focuses on providing high-fidelity data solutions to enable customers to derive insights quickly.

Team Overview: We are a passionate team of technologists, data scientists, and analysts with backgrounds in operational intelligence, law enforcement, large multinationals, and cybersecurity operations. We obsess about designing products that will change the way global companies, governments and nonprofits protect themselves from external threats and global adversaries.

Position Overview: We are looking for a skilled AWS or Azure Data Engineer with a focus on Infrastructure as Code (IaC) and data engineering. Your role will involve designing, implementing, and maintaining scalable, secure cloud infrastructure on AWS or Azure, while also specializing in designing, implementing, and optimizing data pipelines and storage solutions. You will ensure the reliability and availability of infrastructure for data processing on a petabyte scale and collaborate closely with our infrastructure engineers and data engineers to automate and streamline data processes.

Responsibilities

• Design, implement, and maintain scalable, secure AWS or Azure infrastructure with a focus on Infrastructure as Code (IaC).
• Develop and maintain scalable data pipelines using Spark, Scala, Golang, and Python.
• Implement and manage Iceberg tables, ensuring efficient data storage and retrieval.
• Optimize data storage solutions, including hot, cold, and glacier storage tiers.
• Develop and enforce data retention policies and ensure compliance with data governance standards.
• Collaborate with software engineers to ensure the infrastructure effectively supports application requirements.
• Ensure data security and implement necessary measures to protect sensitive information.
• Monitor and troubleshoot data pipelines and infrastructure to ensure high availability and performance.
• Document infrastructure design, data engineering processes, and maintain comprehensive documentation.

Required Qualifications

• Must live in United Kingdom and able to travel to West Midlands area.
• 3+ years of experience in AWS/Azure Infrastructure and/or SRE roles, with applicable Associate or Professional AWS or Azure certificates being a plus.
• Strong background in Infrastructure as Code (IaC) using tools such as Pulumi and Terraform.
• Experience with data orchestration tools such as Dagster, Airflow, Prefect, or Temporal.
• Proficiency in Spark, Scala, Pulumi, and Python. Nice to have Golang.
• Experience with Postgres, GraphQL, and other data manipulation tools.
• Knowledge of big data tools and environments, including Apache Iceberg and other datalake concepts.
• Familiarity with geospatial data formats such as Parquet/GeoParquet, GeoJSON, and Shapefiles.
• Experience with DevOps tools such as Git, Docker, Jenkins, etc.
• Knowledge of networking concepts such as IP Protocol, DNS, and load balancing.
• Excellent problem-solving skills and the ability to think quickly in a high-performance environment.
• Effective communication skills to convey technical concepts to both technical and non-technical stakeholders.

Compensation and Benefits

• Competitive salary
• Comprehensive health, dental, and vision insurance plans
• Flexible hybrid work environment
• Additional benefits like flexible hours, work travel opportunities, competitive vacation time and parental leave

While meeting all of these criteria would be ideal, we understand that some candidates may meet most, but not all. If you\'re passionate, curious and ready to "work smart and get things done," we\'d love to hear from you.