Data Engineer

emagine; Leiria; Teletrabalho; EUR 30 000 - 50 000; Tempo integral

Resumo da oferta

Join a leading e-commerce company specializing in fashion and home décor as a Data Engineer. You'll contribute to impactful data engineering projects, developing solutions for data processing and cloud storage using the latest technologies. Ideal candidates will have strong skills in PySpark and Terraform, and thrive in a remote work environment.

Qualificações

• 1 year of experience with Terraform.
• Proficient in PySpark and Spark SQL for data processing.
• Strong understanding of ETL/ELT processes and data warehousing.

Responsabilidades

• Design and deploy efficient data pipelines according to project requirements.
• Analyze technical challenges and prepare test cases for deployment.
• Document processes and conduct code reviews for quality assurance.

Conhecimentos

PySpark

Spark SQL

Terraform

GitOps

data processing

data warehousing

data modeling

ETL processes

English Fluency

problem-solving

Ferramentas

Databricks

Apache Airflow

Azure Data Lake Storage

Kubernetes

Apache Kafka

Vault

Descrição da oferta de emprego

Industry: E-commerce (Fashion & Home)

Location: Portugal

Work Model: 100% Remote

Start Date: ASAP

Project Language: English

Project Overview:

Join a leading e-commerce company in fashion and home décor, as they undertake critical data engineering projects to support their evolving business needs. This is an exciting opportunity to contribute to high-impact data solutions for an innovative, customer-focused brand, leveraging the latest in data engineering best practices.

Responsibilities:

• Project Understanding and Communication:
• Analyze business and technical challenges from a user perspective.
• Collaborate with Data Architects and Project Managers to ensure solutions align with the client´s data architecture.
• Data Pipeline Development:
• Design, build, and deploy efficient data pipelines according to project requirements.
• Apply best practices for performance, scalability, and maintainability.
• Use Terraform to deploy and manage infrastructure efficiently.
• Testing and Deployment:
• Define test cases and conduct testing in collaboration with the Project Manager.
• Present completed developments to Data Architects and Lead DataOps, ensuring smooth deployment and active monitoring post-deployment.
• Documentation and Peer Review:
• Document processes, tests, and results thoroughly.
• Conduct peer reviews and participate in code reviews for quality assurance.
  

Requirements:

Hard Skills:

• Proficiency in PySpark and Spark SQL for data processing.
• Experience with Databricks and Delta Live Tables for ETL and workflow orchestration.
• Familiarity with Azure Data Lake Storage for data storage and management.
• At least 1 year of experience with Terraform and GitOps practices for infrastructure deployment.
• Strong understanding of ETL/ELT processes, data warehousing, data lakes, and data modeling.
• Knowledge of orchestration tools (e.g., Apache Airflow) for pipeline scheduling and management.
• Experience with data partitioning and lifecycle management in cloud storage.
  
  

Optional: Experience with Databricks Asset Bundles, Kubernetes, Apache Kafka, and Vault is a plus.

Soft Skills:

• English Fluency: Strong written and verbal English skills in a working environment.
• Communication: Ability to convey technical concepts effectively and understand user needs.
• Organizational Skills: Detail-oriented with the ability to maintain structured documentation.
• Problem-Solving: Proactive approach to understanding and addressing data challenges.