Senior Data Engineer | Real-Time & Streaming

Alongside; Porto; Hybrid; EUR 70,000 - 90,000; Full time

Job summary

Alongside is seeking a Senior Data Engineer to design, build, and operate real-time data products on an enterprise platform for a client in financial markets. You will own streaming pipelines from ingestion to APIs and services, guiding production operations and architecture.

The role requires hands-on expertise with Flink, Kafka, AWS, and CI/CD, plus strong Java and Python (PyFlink/Scala) skills. Porto-based hybrid work is offered, with cross-functional collaboration and mentorship

Qualifications

• 7+ years of professional data engineering experience.
• Strong hands-on experience with Apache Flink in production, including DataStream APIs and Flink SQL.
• Strong Java skills, with experience in Python/PyFlink or Scala also valued.
• Solid experience with Apache Kafka, event-driven architectures, and schema management using Avro and Schema Registry.
• Proven experience building APIs and microservices using REST and OpenAPI.
• Strong AWS experience including MSK, Kinesis, Managed Service for Apache Flink, Lambda, API Gateway, ECS/EKS, S3, and IAM.
• Strong SQL and data modelling skills across streaming and batch workloads.
• Experience with CI/CD tooling such as GitLab CI or GitHub Actions, alongside automated unit, integration, contract, and load testing.
• Experience with observability tools such as CloudWatch, OpenTelemetry, Prometheus, and Grafana.
• Fluency in English, both written and spoken.

Responsibilities

• Design and deliver end-to-end streaming data products using Apache Flink, Apache Kafka, and event-driven architectures.
• Develop stateful stream-processing applications, including windowing, event-time handling, watermarks, checkpointing, and exactly-once processing guarantees.
• Build and maintain resilient ingestion, transformation, and delivery pipelines for real-time and batch workloads.
• Manage Kafka topics, partitioning, consumer groups, schemas, and schema evolution.
• Design and develop REST APIs, streaming interfaces, and microservices with clear contracts, versioning, authentication, and rate limiting.
• Deploy and operate services on AWS, using Infrastructure as Code with Terraform.
• Embed data quality controls, automated testing, observability, and actionable alerting throughout the delivery lifecycle.
• Build and improve CI/CD pipelines, including automated deployment, rollback, and safe release practices.
• Monitor production services against agreed SLAs, investigate incidents, and optimise latency, throughput, reliability, and cost.
• Apply security best practices across authentication, authorisation, encryption, secrets management, and least-privilege access.
• Translate business requirements into technical designs, challenge assumptions, and propose practical solutions.
• Review code and designs, mentor engineers, and contribute to engineering standards and technical documentation.

Skills

Streaming data engineering

Apache Flink

Apache Kafka

Java

Python / PyFlink

Scala

Avro / Schema Registry

REST / OpenAPI

AWS

SQL / data modelling

CI/CD tooling

Observability tools

英语沟通

Tools

Terraform

Kafka schema management

GitLab CI / GitHub Actions

CloudWatch / OpenTelemetry / Prometheus / Grafana

Job description

Alongside partners with organizations to drive digital transformation and build high-performing tech teams.

We are looking for a Senior Data Engineer to join a project with one of our clients operating in the financial markets industry.

You will design, build, and operate real-time data products on an enterprise data platform, taking ownership from event ingestion and stream processing to the APIs and services used by consumers. This is a hands-on role combining advanced streaming engineering, production ownership, and technical guidance.

Responsabilities:

• Design and deliver end-to-end streaming data products using Apache Flink, Apache Kafka, and event-driven architectures.
• Develop stateful stream-processing applications, including windowing, event-time handling, watermarks, checkpointing, and exactly-once processing guarantees.
• Build and maintain resilient ingestion, transformation, and delivery pipelines for real-time and batch workloads.
• Manage Kafka topics, partitioning, consumer groups, schemas, and schema evolution.
• Design and develop REST APIs, streaming interfaces, and microservices with clear contracts, versioning, authentication, and rate limiting.
• Deploy and operate services on AWS, using Infrastructure as Code with Terraform.
• Embed data quality controls, automated testing, observability, and actionable alerting throughout the delivery lifecycle.
• Build and improve CI/CD pipelines, including automated deployment, rollback, and safe release practices.
• Monitor production services against agreed SLAs, investigate incidents, and optimise latency, throughput, reliability, and cost.
• Apply security best practices across authentication, authorisation, encryption, secrets management, and least-privilege access.
• Translate business requirements into technical designs, challenge assumptions, and propose practical solutions.
• Review code and designs, mentor engineers, and contribute to engineering standards and technical documentation.

• 7+ years of professional experience in data engineering, including at least 3 years building and operating real-time or streaming systems in production.
• Strong hands-on experience with Apache Flink in production, including DataStream APIs and Flink SQL.
• Strong Java skills, with experience in Python/PyFlink or Scala also valued.
• Solid experience with Apache Kafka, event-driven architectures, and schema management using Avro and Schema Registry.
• Proven experience building APIs and microservices using REST and OpenAPI.
• Strong AWS experience, including relevant services such as MSK, Kinesis, Managed Service for Apache Flink, Lambda, API Gateway, ECS/EKS, S3, and IAM.
• Strong SQL and data modelling skills across streaming and batch workloads.
• Experience with CI/CD tooling such as GitLab CI or GitHub Actions, alongside automated unit, integration, contract, and load testing.
• Experience with observability tools such as CloudWatch, OpenTelemetry, Prometheus, and Grafana.
• Fluency in English, both written and spoken.

• The opportunity to build production-grade real-time data products within the financial markets industry.
• End-to-end ownership across streaming pipelines, APIs, and production operations.
• Collaboration with platform engineers, product owners, and business stakeholders.
• Scope to influence technical decisions, improve engineering practices, and mentor other engineers.
• Hybrid working model in Porto.