Data Engineer

emagine; Leiria; Remote; EUR 45,000 - 75,000; Full time

Job summary

emagine is seeking a data engineer to drive data pipeline development for an e-commerce fashion and home décor brand. The role focuses on PySpark, Databricks, Delta Live Tables, and orchestration, with a remote-first setup and ASAP start.

You will collaborate with data architects and project managers to deliver scalable data solutions and robust testing processes. Ideal candidates will have Terraform/GitOps experience, familiarity with Azure Data Lake Storage, and strong English communication.

Qualifications

• Minimum 1 year of experience with Terraform and GitOps practices for infrastructure deployment.
• Proficiency with PySpark and Spark SQL for data processing.
• Experience with Databricks and Delta Live Tables for ETL and orchestration.
• Familiarity with Azure Data Lake Storage for data storage and management.

Responsibilities

• Analyze business and technical challenges from a user perspective.
• Design, build, and deploy data pipelines with performance and scalability in mind.
• Collaborate with Data Architects and Project Managers to align solutions with data architecture.
• Document processes, tests, and results; participate in code and peer reviews.

Skills

English Fluency

Communication

Organizational Skills

Problem-Solving

Education

Bachelor's degree in a related field

Tools

Databricks

Delta Live Tables

Azure Data Lake Storage

Apache Airflow

Terraform

GitOps

Kubernetes

Apache Kafka

Vault

Job description

Industry: E-commerce (Fashion & Home)

Location: Portugal

Work Model: 100% Remote

Start Date: ASAP

Project Language: English

Project Overview

Join a leading e-commerce company in fashion and home décor, as they undertake critical data engineering projects to support their evolving business needs. This is an exciting opportunity to contribute to high-impact data solutions for an innovative, customer-focused brand, leveraging the latest in data engineering best practices.

Responsibilities

• Project Understanding and Communication:
  • Analyze business and technical challenges from a user perspective.
  • Collaborate with Data Architects and Project Managers to ensure solutions align with the client´s data architecture.
• Data Pipeline Development:
  • Design, build, and deploy efficient data pipelines according to project requirements.
  • Apply best practices for performance, scalability, and maintainability.
  • Use Terraform to deploy and manage infrastructure efficiently.
• Testing and Deployment:
  • Define test cases and conduct testing in collaboration with the Project Manager.
  • Present completed developments to Data Architects and Lead DataOps, ensuring smooth deployment and active monitoring post-deployment.
• Documentation and Peer Review:
  • Document processes, tests, and results thoroughly.
  • Conduct peer reviews and participate in code reviews for quality assurance.

Requirements

Hard Skills:

• Proficiency in PySpark and Spark SQL for data processing.
• Experience with Databricks and Delta Live Tables for ETL and workflow orchestration.
• Familiarity with Azure Data Lake Storage for data storage and management.
• At least 1 year of experience with Terraform and GitOps practices for infrastructure deployment.
• Strong understanding of ETL/ELT processes, data warehousing, data lakes, and data modeling.
• Knowledge of orchestration tools (e.g., Apache Airflow) for pipeline scheduling and management.
• Experience with data partitioning and lifecycle management in cloud storage.

Optional: Experience with Databricks Asset Bundles, Kubernetes, Apache Kafka, and Vault is a plus.

Soft Skills:

• English Fluency: Strong written and verbal English skills in a working environment.
• Communication: Ability to convey technical concepts effectively and understand user needs.
• Organizational Skills: Detail-oriented with the ability to maintain structured documentation.
• Problem-Solving: Proactive approach to understanding and addressing data challenges.