Data Engineer (UK)

Atreides Caseri Inc; Cambridgeshire and Peterborough; Hybrid; GBP 50,000 - 70,000; Full time

Job summary

A dynamic data solutions company in the UK is looking for a skilled Data Engineer to manage extensive datasets and develop data pipelines. This hybrid role requires collaboration with clients while ensuring data security and integrity. Candidates should have proficiency in Python, Java, and Azure technologies, alongside a background in geospatial data handling. Competitive salary and comprehensive benefits offered.

Benefits

Competitive salary

Comprehensive health insurance

Flexible hybrid work environment

Parental leave

Qualifications

• 5+ years of experience in data engineering or related roles.
• Strong problem-solving skills in a high-performance environment.
• Ability to communicate technical concepts effectively.

Responsibilities

• Design, develop, and maintain scalable data pipelines.
• Create geospatial data products for visualization tools.
• Ensure compliance with data governance standards.

Skills

Python

Java

Golang

SQL

Data pipeline development

Data security

Geospatial data handling

Azure Data Factory

Tools

Spark

Azure

Airflow

Job description

Company Overview: Atreides helps organizations transform large and complex multi-modal datasets into information-rich, geo-spatial data subscriptions that can be used across a wide spectrum of use cases. Currently, Atreides focuses on providing defence intelligence professionals with high-fidelity data solutions that enable these customers to derive insights in minutes. The company is a fast-moving, fast-growing, high-performance startup.

We value a diverse team and believe inclusion performs better. We believe that trust + autonomy = better results and more job satisfaction. Since we are building something new and helping unlock the power of massive-scale data to make the world safer, stronger, and more prosperous, a mission-driven mindset and entrepreneurial spirit are a must.

Team Overview: We are a passionate team of technologists, data scientists, and analysts with backgrounds in operational intelligence, law enforcement, large multinationals, and cybersecurity operations. We obsess about designing products that will change the way global companies, governments and nonprofits protect themselves from external threats and global adversaries.

Position Overview: We're looking for a skilled Data Engineer to work on-site with a key customer, specialising in designing, implementing, and optimising data pipelines and storage solutions to manage our extensive datasets, ranging from hundreds of terabytes to petabytes. You will play a key role in ensuring data integrity, accessibility, and security while working closely with customer and partner teams, while supported by the wider Atreides team.

This role is hybrid, based one day per week on-site with an Atreides client based in Cambridgeshire, UK.

Team Principles:

At Atreides, we believe that teams work best when they:

• Remain curious and passionate in all aspects of our work
• Promote clear, direct, and transparent communication
• Embrace the 'measure twice, cut once' philosophy
• Value and encourage diverse ideas and technologies
• Lead with empathy in all interactions

Responsibilities:

• Design, develop, and maintain scalable data pipelines, including from external API's or datasets and fuse them with other data sources.
• Create geospatial data products to be used with visualision tools to facilitate understanding with Analysts and End Users
• Develop and enforce data retention policies and ensure compliance with data governance standards
• Work with a wide range of data types, applying knowledge of integrating different schemas into a single unified schema
• Collaborate with infrastructure engineers to integrate data processes with Azure infrastructure and automation tools
• Ensure data security and implement necessary measures to protect sensitive information
• Monitor and troubleshoot data pipelines to ensure high availability and performance
• Document data engineering processes and maintain comprehensive documentation

Desired Skills, Qualifications, Experience:

• 5+ years of experience in data engineering or related roles
• Proficiency in Python, Java, Golang, Spark, Azure, Synapse Analytics, SQL, Datalake Technology, PySpark, Python, Azure Functions, Azure Data Factory Nice to have: Scala.
• Familiarity with Kepler geospatial visualisation tool, or other similar tools, and experience working with geospatial data would also be an advantage
• Knowledge of a wide range of data types and knowledge of integrating different schemas into a single unified schema.
• Strong understanding of data optimization, storage solutions, and data retention policies
• Familiarity with Azure infrastructure and site reliability engineering principles. Azure Data Eng/DevOps Associate or Professional Azure certificates are a plus.
• Knowledge of big data tools and environments, having previously worked with at least 100TB of data at production scale
• Familiarity with geospatial data formats such as Parquet/GeoParquet, GeoJSON, and Shapefiles is a plus.
• Experience with data orchestration tools such as Airflow, Prefect, Dagster, or Temporal, would also be a plus.
• Excellent problem-solving skills and the ability to think quickly in a high-performance environment
• Effective communication skills to convey technical concepts to both technical and non-technical stakeholders
• Ability to work in an Agile team and be self-motivating
• Ability to work independently on a customer-facing project, with minimal supervision
• Ability to travel to the customer site (near Huntingdon, Cambridgeshire), currently once a week, this could be reduced with agreement with the customer
• SC clearance

Compensation and Benefits:

• Competitive salary
• Comprehensive health, dental, and vision insurance plans
• Flexible hybrid work environment
• Additional benefits like flexible hours, work travel opportunities, competitive vacation time and parental leave

You must have the right to work in the United Kingdom. Please note that we do not provide visa sponsorship.

This position requires eligibility to obtain a UK SC government security clearance, which typically requires residency or other qualifying criteria.

While meeting all of these criteria would be ideal, we understand that some candidates may meet most, but not all. If you're passionate, curious and ready to "work smart and get things done," we'd love to hear from you.