Senior Data Engineer

HITO Solutions; Matosinhos; Presencial; EUR 50 000 - 70 000; Tempo integral

Resumo da oferta

Uma empresa em crescimento na área de soluções tecnológicas está buscando um Tech Lead para Engenharia de Dados. O profissional será responsável por gerenciar projetos de dados, orientar uma equipe júnior e garantir a implementação de soluções robustas em um ambiente baseado em Azure. A oportunidade oferece um ambiente dinâmico e desafiador para profissionais com experiência em engenharia de dados e habilidades avançadas em ferramentas como Databricks e Azure Data Factory.

Qualificações

• 5+ anos de experiência em engenharia de dados em ambientes de grande escala.
• Proficiente em ecossistemas de nuvem, especialmente Azure.
• Conhecimento em práticas sólidas de engenharia de software.

Responsabilidades

• Atuar como contato técnico principal para partes interessadas e supervisionar a execução de projetos de dados.
• Mentorar engenheiros juniores e liderar discussões de design.
• Desenvolver pipelines de dados em lotes e em tempo real.

Conhecimentos

Python

SQL

PySpark

Databricks

Formação académica

Mestrado ou PhD em Engenharia, Ciência da Computação

Ferramentas

Azure Data Factory

Azure DevOps

Terraform

Descrição da oferta de emprego

Position Overview:
As a Tech Lead Data Engineering, you will act as the key technical advisor for stakeholders, take ownership of data engineering projects, manage technical debt, and mentor junior team members. You will contribute to architecture, design, and development of robust data solutions.

Key Responsibilities

• Serve as the primary technical contact for stakeholders and oversee data project execution.
• Mentor junior engineers and lead design discussions.
• Develop batch and real-time data pipelines using Databricks, PySpark, and DBT.
• Clean, transform, and optimize raw data for analytics, ensuring cost and performance efficiency.
• Design and manage database schemas and optimize storage formats (e.g., Parquet, Delta Lake).
• Implement data quality checks, lineage documentation, and advanced partitioning/indexing strategies.
• Partner with data scientists and analysts to ensure seamless data access for analytics and machine learning.
• Work with Platform Engineers to align on best practices.

Security & Compliance:

• Enforce role-based access controls, data encryption, and compliance with data privacy regulations.

Infrastructure & Configuration:

• Manage Azure cloud infrastructure, including Data Lake, Blob Storage, and Databricks clusters.
• Build and maintain pipelines using Azure Data Factory.
• Maintain detailed documentation of pipelines, processes, and tools.
• Propose and implement optimizations for systems and workflows.

Education & Experience

• Masters or PhD in Engineering, Computer Science, or related fields.
• 5+ years of experience in data engineering, particularly in large-scale environments.
• Proficiency with cloud ecosystems, especially Azure.

Technical Skills

Programming & Tools:

• Advanced Python (e.g., Pandas, dependency management) and SQL.
• Strong software engineering practices (e.g., design patterns, clean code, testing).

Data Processing & Storage:

• Expertise in PySpark, Delta Lake, and storage formats (e.g., Parquet, JSON).
• Proficiency with Azure Data Factory, Databricks, and database management (SQL and NoSQL).
• Strong skills in schema design, ETL workflows, and modeling (Dimensional, 3NF, Data Vault).

DevOps & Automation:

• Experience with CI/CD tools (Azure DevOps) and Infrastructure as Code (ARM templates, Terraform preferred).

Security & Monitoring:

• Knowledge of role-based access, encryption, and monitoring tools (e.g., Azure Monitor).