Senior Data Engineer - MLOps & Cloud Data Pipelines

emagine; Lisboa; Presencial; EUR 65 000 - 100 000; Tempo integral

Resumo da oferta

emagine is expanding its data engineering team in Lisbon, Portugal, seeking a professional focused on advanced data engineering, DevOps foundations, and MLOps knowledge. You will design, optimize, and maintain infrastructure and pipelines supporting local data projects, data science work, and analytics, acting as a bridge between model development and large-scale production.

Key responsibilities include building robust ingestion pipelines, organizing BigQuery storage with proper environment

Qualificações

• Experience in data engineering for large-scale data systems.
• GCP BigQuery proficiency and cloud data architecture knowledge.
• Experience designing automated CI/CD pipelines (GitHub Actions preferred).
• Experience with monorepo architectures and modern data lake/lakehouse patterns.
• Fluent or advanced English (B2/C1).

Responsabilidades

• Develop, monitor, and automate robust data ingestion pipelines (batch/streaming).
• Design and organize data storage in BigQuery with proper environment segregation.
• Support data science by enabling testing, packaging, deployment, and monitoring of models.
• Ensure automated code lifecycle and CI/CD implementations.
• Contribute to scalable, secure, well-documented monorepo-based solutions.

Conhecimentos

GCP BigQuery

Docker

Kubernetes

CI/CD

Monorepo

Data lake/Lakehouse

English (B2/C1)

Ferramentas

Datadog

Grafana

Descrição da oferta de emprego

• Context and Mission of the Position
  
  

We are expanding our data engineering structure and are seeking a professional with a technical profile, focused on advanced data engineering, solid foundations in DevOps, and knowledge of MLOps. This individual will be responsible for designing, optimizing, and maintaining the infrastructure and pipelines that support local data projects, data science projects, and the analytical operations of the organization, serving as a critical bridge between model development and large-scale production.

• Main Responsibilities and Activities
• Complex Pipeline Engineering: Develop, monitor, and automate robust data ingestion pipelines (Batch and Streaming), handling complex scenarios such as advanced integration of third-party APIs and web scraping of application APIs.
• Data Warehouse Architecture and Conventions: Design and organize data storage in Google BigQuery using best modeling practices. Ensure the correct transition and logical segregation of environments, from raw data reception (Data Swamp / Data Lake) to optimized availability layers (Publish Layers).
• Machine Learning Operationalization (MLOps): Actively support the Data Science stack, creating the technical bases necessary for autonomy, local testing, product packaging, deployment, and continuous monitoring of predictive models in a production environment.
• Automation and Infrastructure (CI/CD): Ensure the code lifecycle and that implementations are automated.
• Software Engineering Culture: Contribute to highly scalable, secure, and well-documented solutions. Promote and structure modern repositories in a Monorepo environment, facilitating collaboration and independence of Data Science teams.
• Technical Profile and Candidate Requirements
  
  

Mandatory Requirements (Hard Skills)

• Consolidated experience in Data Engineering or Software Engineering focused on large-scale data systems.
• Practical experience in the cloud ecosystem GCP (Google Cloud Platform), with a specialized focus on BigQuery.
• Practical experience with containerization and orchestration tools: Docker and Kubernetes (K8s).
• Proven proficiency in designing automated CI/CD pipelines using preferably GitHub Actions (or equivalent market tools).
• Advanced practical knowledge in organizing and managing code in Monorepo architectures.
• Strong understanding of modern data architectures (Lakehouse, Data Lake) and their respective data layer conventions.
• Proficiency in English: Minimum intermediate-advanced level (e.g., B2 or C1) for communication in technical and international environments.
  
  

Valued Requirements (Differentials)

• Direct practical experience in MLOps concepts and tools (model lifecycle management, model serving APIs, feature stores).
• Experience in deploying solutions in orchestrated clusters (Kubernetes), depending on architectural needs.
• Experience with Observability Using Logging tools like Datadog or Grafana
• Summary of Required Tech Stack
  
  

Key Technologies / Concepts

Cloud & Storage

Google Cloud Platform (GCP), Google BigQuery

Infrastructure & DevOps

Kubernetes (K8s), Docker, Virtual Machines (VMs)

CI/CD & Repositories

GitHub Actions, Monorepo

ML Methodologies

MLOps (Operationalization and Deployment of Data Science Models)

Data Architecture

Data Swamp, Data Lake, Publish Layers, DW Conventions

Languages

English (B2 or C1)