Remote Data Engineer: Spark, Databricks & ETL

emagine; Leiria; Presencial; EUR 42 000 - 66 000; Tempo integral

Resumo da oferta

emagine is hiring for a data engineering role focusing on cloud-based ETL/ELT pipelines in a 100% remote setup for Portugal. The project uses PySpark, Databricks, Delta Live Tables, and Azure Data Lake Storage to build scalable data solutions for an e‑commerce fashion & home décor brand. English is the working language.

ASAP start. The candidate will design, implement, and validate data pipelines, collaborate with architects and PMs, and implement Terraform-driven infrastructure, ensuring robust

Qualificações

• Proficient in PySpark and Spark SQL for large-scale data processing.
• Experience with Databricks and Delta Live Tables for ETL workflows.
• Knowledge of Azure Data Lake Storage for storage and governance.
• At least 1 year of Terraform and GitOps practices.
• Strong understanding of ETL/ELT, data warehousing, data lakes, and data modeling.
• Familiarity with orchestration tools like Apache Airflow for scheduling.
• Experience with data partitioning and lifecycle management in cloud storage.

Responsabilidades

• Analyze business and technical challenges from a user perspective.
• Collaborate with Data Architects and Project Managers to ensure solutions align with the client´s data architecture.
• Design, build, and deploy efficient data pipelines according to project requirements.
• Apply best practices for performance, scalability, and maintainability.
• Use Terraform to deploy and manage infrastructure efficiently.
• Define test cases and conduct testing in collaboration with the Project Manager.
• Present completed developments to Data Architects and Lead DataOps, ensuring smooth deployment and active monitoring post-deployment.
• Document processes, tests, and results thoroughly.
• Conduct peer reviews and participate in code reviews for quality assurance.

Conhecimentos

PySpark / Spark SQL

Databricks & Delta Live Tables

Azure Data Lake Storage

Terraform & GitOps

ETL / ELT concepts

Airflow orchestration

Data partitioning & lifecycle mgmt

Ferramentas

Apache Airflow

Kubernetes

Kafka

Vault

Descrição da oferta de emprego

emagine is hiring for a data engineering role focusing on cloud-based ETL/ELT pipelines in a 100% remote setup for Portugal. The project uses PySpark, Databricks, Delta Live Tables, and Azure Data Lake Storage to build scalable data solutions for an e‑commerce fashion & home décor brand. English is the working language.

ASAP start. The candidate will design, implement, and validate data pipelines, collaborate with architects and PMs, and implement Terraform-driven infrastructure, ensuring robust