Senior Data Engineer

QUANTEAM (RAINBOW PARTNERS Group); Porto; On-site; EUR 45,000 - 60,000; Full time

Benefits offered by this job

Health Insurance

Extra holiday: Christmas Eve or New...

Access to Quanteam Academy

Elevate Program

Ongoing consultant support

Job summary

Quanteam, founding entity of RAINBOW PARTNERS, is seeking a Senior Data Engineer to join the Porto team. The role focuses on designing, building and operating real-time data products and streaming services using Apache Flink and Kafka, deployed on AWS with IaC.

The ideal candidate has 7+ years in Data Engineering, with 3+ years in production real-time systems, strong Java/Python skills, and fluency in English.

Qualifications

• Bachelor's degree or equivalent experience.
• 7+ years in Data Engineering, with 3+ years in real-time/streaming systems.
• Strong hands-on with Flink in production (Java, Scala, or Python).
• Extensive Kafka experience and event-driven architectures.

Responsibilities

• Design, build and operate real-time data products from ingestion to APIs.
• Develop stateful streaming apps using Flink (windowing, event-time, watermarks).
• Build and manage Kafka-based streams and schemas.
• Design REST and streaming APIs and microservices.
• Deploy on AWS with containerization and IaC (Terraform).
• Improve CI/CD pipelines and automated testing.
• Provide production support and runbooks.

Skills

Real-time data

Streaming systems

SQL

Data modeling

English language

Education

BSc in Computer Science/Engineering

Tools

Apache Flink

Apache Kafka

Java

Python

Terraform

AWS (ECS/EKS, Lambda, S3)

GitLab CI / GitHub Actions

Job description

Quanteam, founding entity of RAINBOW PARTNERS, is a consulting firm specialized in Banking and Financial Services. With more than 1200 consultants across 10 countries, we combine business and IT expertise to support leading institutions in transformation projects.

Our core areas:

Operations & Finance

We are looking for a Senior Data Engineer to join our team in Porto.

• Design, build and operate real-time data products, from event ingestion to consumer-facing APIs.
• Develop stateful stream-processing applications using Apache Flink (DataStream API and Flink SQL), including windowing, event-time processing, watermarks, checkpointing and exactly-once delivery.
• Build and manage Kafka-based event streams, including topic design, schema management, schema evolution, partitioning and consumer groups.
• Design and implement REST and streaming APIs (e.g. WebSockets and Server-Sent Events) and the microservices behind them.
• Develop APIs with clear contracts, versioning, authentication and rate-limiting mechanisms.
• Deploy and operate services on AWS using containers, ECS/EKS, Lambda, API Gateway and Infrastructure as Code with Terraform.
• Optimize streaming jobs and services to meet latency, throughput, availability and cost targets.
• Integrate data products with the Group Data Platform, following data contracts, catalogue and access policies.
• Implement data quality, observability and alerting mechanisms, including metrics, logs, tracing, consumer lag and data freshness monitoring.
• Develop and improve CI/CD pipelines, automated testing and safe release practices.
• Provide production support, investigate incidents and implement permanent solutions.
• Produce technical documentation, solution designs and operational runbooks.
• Conduct code and technical design reviews, mentor engineers and promote real-time engineering best practices.
• Collaborate with platform teams, product owners and business stakeholders, challenging requirements and proposing efficient technical solutions.
• Bachelor's degree in Computer Science, Engineering, Information Systems or a related field, or equivalent practical experience.
• At least 7 years of experience in Data Engineering, including a minimum of 3 years building and operating real-time or streaming systems in production.
• Strong hands‑on experience with Apache Flink in production, using Java, Scala or Python/PyFlink.
• Strong experience with Apache Kafka and event‑driven architectures, including schema management (Avro, Schema Registry).
• Proven experience designing and developing REST APIs and microservices using Java and/or Python. Experience with streaming APIs is a plus.
• Solid experience with AWS services, particularly MSK, Kinesis, Managed Service for Apache Flink, Lambda, API Gateway, ECS/EKS, S3 and IAM.
• Experience with Terraform and Infrastructure as Code concepts.
• Knowledge of CI/CD practices and tools such as GitLab CI, GitHub Actions or equivalent.
• Strong SQL and data modelling skills, covering both streaming and batch data.
• Experience with automated testing, including unit, integration, contract and load testing.
• Knowledge of observability tools such as CloudWatch, OpenTelemetry, Prometheus and Grafana.
• Understanding of security principles for data services, including OAuth2/OIDC, authentication, authorisation, encryption and least‑privilege access.
• Experience operating production services against SLAs and troubleshooting complex technical issues.
• Strong communication skills, with the ability to lead technical discussions and collaborate with engineering teams and business stakeholders.
• Strong analytical and problem‑solving skills, with a proactive ownership mindset.
• Fluency in English is mandatory.
• Based in Portugal, with availability to work in Porto.
• This role offers a gross annual salary between 45.000€ to 60.000€, defined according to the candidate’s experience and competencies
• Permanent contract (CTI).
• Health Insurance.
• Extra holiday: Christmas Eve or New Year's Eve.
• Access to Quanteam Academy for continuous learning and certifications.
• Elevate Program to support professional development and career growth.
• Ongoing follow‑up and technical support provided by a Consultant Expert