Data Engineer

Stott and May; Portugal; Híbrido; EUR 54 000 - 66 000; Tempo integral

Vantagens oferecidas por esta oferta de emprego

Health insurance

Mental health support

Professional development funding

Hybrid/Remote work

Resumo da oferta

Stott and May is seeking a Data Engineer to design, build and optimise data flows for a cloud-based platform. The role focuses on ingestion, transformation and modelling with Snowflake at the core and Kafka for real-time data capture.

You will work on KSQL/KStreams pipelines, data models in Snowflake and ensure data quality across the lifecycle, with hybrid/remote options and opportunities across Portugal.

Qualificações

• Solid commercial experience as a Data Engineer.
• Experience designing, building and optimising ELT/ETL pipelines.
• Strong SQL and Python or another scripting language.

Responsabilidades

• Design, build and maintain scalable ELT/ETL data pipelines within Snowflake.
• Implement data ingestion from various sources, including real-time streams using Kafka.
• Develop and deploy KSQL and KStreams pipelines for real-time processing.
• Develop efficient Snowflake data models for analytics.
• Ensure data quality, integrity and reliability across the data lifecycle.
• Optimise pipelines and Snowflake workloads for performance and scalability.
• Collaborate with architects, analysts and data scientists to deliver data solutions.
• Contribute to data engineering standards and best practices.

Conhecimentos

SQL

Python

Kafka

Airflow

Snowflake

CI/CD

Ferramentas

Snowflake

Kafka

KSQL

KStreams

Airflow

Confluent Schema Registry

Descrição da oferta de emprego

Location: Portugal or Spain (Hybrid/Remote)

Salary: Up to €60,000

About the Company

We are working with an independent technology consultancy that works across data, cloud and technology transformation, helping clients modernise their platforms and make better use of their data.

About the Role

As part of a major data modernisation programme, you will be responsible for designing, building and optimising the data flows that power a new cloud-based platform.

The role will focus heavily on data ingestion, transformation and modelling, with Snowflake as the core data warehouse and Kafka supporting real-time data capture and processing.

This is an opportunity for a Data Engineer who enjoys working with real-time data and solving technical challenges around streaming, integration and large-scale data transformation.

Responsibilities

• Design, build and maintain scalable ELT/ETL data pipelines within Snowflake.
• Implement data ingestion from a range of sources, including real-time streams using Kafka.
• Develop and deploy KSQL and KStreams pipelines for real-time data processing.
• Develop efficient and scalable data models within Snowflake to support analytics and business intelligence.
• Ensure data quality, integrity and reliability throughout the data lifecycle.
• Optimise data pipelines and Snowflake workloads for performance and scalability.
• Work closely with architects, analysts and data scientists to understand requirements and deliver effective data solutions.
• Contribute to data engineering standards, best practices and continuous improvement across the platform.
• Solid commercial experience as a Data Engineer, with a track record of delivering cloud-based data solutions.
• Previous experience within insurance or financial services.
• Strong knowledge of Snowflake, including data modelling and performance optimisation.
• Hands-on experience with Kafka for data ingestion and stream processing.
• Practical experience with KStreams and KSQL pipeline development and deployment.
• Strong SQL skills and experience with Python or another scripting language.
• Experience with data orchestration tools such as Airflow.
• Experience working with at least one major cloud platform, such as AWS, Azure or GCP.
• Strong understanding of data ingestion, transformation, modelling and integration.
• Ability to work collaboratively with technical and non-technical stakeholders.
• Experience building data pipelines for Policy Administration Systems, particularly Guidewire.
• Understanding of data schemas and Confluent Schema Registry.
• Experience working with CI/CD practices and DevOps teams.
• Broader experience across AWS, Azure or GCP.
• Salary up to €60,000.
• Health insurance and mental health support.
• Financial support for ongoing professional training and development.
• Flexible working, with hybrid working available for Lisbon-based employees and remote working available elsewhere in Portugal.
• Opportunities to work on modern data and cloud transformation programmes.
• Career development opportunities within a growing technology consultancy.