Data Engineer: Build Scalable Pipelines for Ocean Tech

Ocean Infinity; Porto; Sur place; EUR 50 000 - 61 000; Plein temps

Résumé du poste

Ocean Infinity in Porto is seeking a Data Engineer to build scalable data pipelines and automation that transform complex workflows into production-grade data systems. You will collaborate with data analysts, product teams, and AI engineers to deliver robust datasets powering analytics and operational systems.

The role focuses on Bronze/Silver/Gold data transformations, Python-based processing, cloud platforms, data lakes, and CI/CD for data workflows.

Qualifications

• A degree in Computer Science, Mathematics, Engineering, or a related field, or equivalent practical experience.
• Minimum 2 years of experience as a Data Engineer, Backend Engineer, or Software Engineer working with data-intensive systems and data pipelines.
• Solid software engineering skills in Python, with experience writing reliable, maintainable data processing code and automation workflows.
• Good SQL skills, with an understanding of query optimization and working with analytical datasets.
• Experience building and maintaining data pipelines and transformation workflows, ideally in cloud-based environments.
• Hands-on experience with at least one cloud platform (e.g., AWS, GCP, or Azure), particularly for building and running data pipelines, storage systems, and compute workloads.
• Familiarity with data lake and lakehouse environments, including object storage systems (e.g., S3 or equivalent) and structured transformation layers (e.g., Bronze/Silver/Gold patterns).
• Understanding of unstructured and semi-structured data storage patterns (e.g., JSON, logs, event data, files in object storage) and how to process them efficiently.
• Experience with relational and/or NoSQL databases (e.g., Postgres, MongoDB, Redis), including schema design and indexing.
• Working knowledge of containerization technologies such as Docker.
• Good understanding of data modelling principles and how to structure datasets for analytics and downstream consumption.
• Exposure to distributed systems concepts and large-scale data processing patterns, with emphasis on practical implementation and debugging.
• Strong problem-solving skills with a focus on reliability, performance, and automation in production systems.
• Ability to collaborate effectively with analysts, product teams, and engineers, translating requirements into robust, testable implementations.
• Interest in improving and refactoring existing pipelines and workflows to increase reliability, maintainability, and automation.
• Comfortable working in modern engineering practices including Git-based workflows, code reviews, CI/CD pipelines, and automated testing for data systems.
• Exposure to workflow orchestration tools such as Airflow, Prefect, or similar, including scheduling, dependency management, retries, and observability.

Responsabilités

• Develop and implement scalable data pipelines that automate manual data processes and improve reliability and efficiency across the organization.
• Build and maintain end-to-end data transformation workflows (Bronze → Silver → Gold), focusing on implementing business logic, data validation, and reusable transformation components.
• Develop robust Python-based data processing solutions, including scripts, services, and automation tools that support ingestion, transformation, and data delivery.
• Implement data integration pipelines across a variety of sources, including operational databases, external APIs, and event/streaming data, using both batch and near-real-time approaches.
• Develop and maintain data quality checks, validation rules, and testing frameworks to ensure accuracy, consistency, and reliability of data products.
• Build backend data services and lightweight APIs that expose curated datasets to analytics tools, reporting systems, and downstream applications.
• Collaborate closely with data analysts, product teams, and AI engineers to translate data requirements into efficient, production-ready implementations.
• Optimize and refactor existing pipelines and processes to improve performance, reliability, and maintainability, with a strong focus on code quality and automation.
• Follow and contribute to engineering best practices, including version control standards, modular pipeline design, CI/CD for data workflows, and documentation of implemented solutions.
• Participate in code reviews and pairing, learning from more experienced engineers and sharing implementation knowledge and reusable patterns with the team.

Connaissances

Python

SQL

Cloud platforms

Data pipelines

Docker

Data modeling

CI/CD

Airflow

Formation

Bachelor's degree in Computer Science or related field

Outils

PostgreSQL

MongoDB

Docker

Kubernetes

Description du poste

Ocean Infinity in Porto is seeking a Data Engineer to build scalable data pipelines and automation that transform complex workflows into production-grade data systems. You will collaborate with data analysts, product teams, and AI engineers to deliver robust datasets powering analytics and operational systems.

The role focuses on Bronze/Silver/Gold data transformations, Python-based processing, cloud platforms, data lakes, and CI/CD for data workflows.