Data Engineer

Zunzun Solutions; Linhares; Presencial; BRL 80.000 - 120.000; Tempo integral

Resumo da oferta

A data solutions company located in Brazil is looking for a skilled Data Engineer specializing in Azure Databricks. This role focuses on designing enterprise-grade data pipelines, optimizing database performance, and implementing REST APIs. Candidates should have extensive experience with Python, SQL Server, and Azure Data Factory. The position offers an opportunity to work on modernizing data platforms while collaborating with cross-functional teams. Relevant certifications in Azure and Databricks are preferred.

Qualificações

• 5+ years of hands-on expertise with Azure Databricks and Python.
• 5+ years of experience with Azure Data Factory for orchestrating pipelines.
• Strong SQL Server experience with a focus on optimization.
• Experience in SSIS package design and performance tuning.

Responsabilidades

• Design and optimize ETL/ELT pipelines using Azure Databricks.
• Develop data flows, transformations with ADF and T-SQL.
• Optimize database performance through SQL query tuning.
• Implement REST APIs to enhance data accessibility.

Conhecimentos

Azure Databricks

Python

PySpark

SQL Server

Data Factory

ETL/ELT processes

Data Governance

Git

REST APIs

Agile/Scrum

Formação académica

Microsoft Certified: Azure Data Engineer Associate (DP‑203)

Databricks Certified Data Engineer Associate

Ferramentas

Azure Data Factory

SQL Server Management Studio

Azure DevOps

Microsoft Purview

Descrição da oferta de emprego

Summary

We are seeking a highly skilled Data Engineer (Azure Databricks) to design, implement, and optimize enterprise‑grade data pipelines. In this role, you will leverage Azure Databricks, Azure Data Factory, SQL Server, and Python to enable scalable, governed, and performant data solutions. You will play a key role in modernizing our data platform on the Azure Cloud, ensuring reliability, efficiency, and compliance across the full data lifecycle.

Key Responsibilities

• Data Pipeline Development: Design, build, and optimize ETL / ELT pipelines using Azure Databricks (PySpark, Delta Lake) and Azure Data Factory (ADF).
• Data Flows & Transformations: Develop pipelines, data flows, and complex transformations with ADF, PySpark, and T‑SQL for seamless data extraction, transformation, and loading.
• Data Processing: Develop Databricks Python notebooks for tasks such as joining, filtering, and pre‑aggregation.
• Database & Query Optimization: Optimize database performance through SQL query tuning, index optimization, and code improvements to ensure efficient data retrieval and manipulation.
• SSIS & Migration Support: Maintain and enhance SSIS package design and deployment for legacy workloads; contribute to migration and modernization into cloud‑native pipelines.
• Collaboration & DevOps: Work with cross‑functional teams using Git (Azure Repos) for version control and Azure DevOps pipelines (CI / CD) for deployment.
• Data Governance & Security: Partner with governance teams to integrate Microsoft Purview and Unity Catalog for cataloging, lineage tracking, and role‑based security.
• API & External Integration: Implement REST APIs to retrieve analytics data from diverse external data feeds, enhancing accessibility and interoperability.
• Automation: Automate ETL processes and database maintenance tasks using SQL Agent Jobs, ensuring data integrity and operational reliability.
• Advanced SQL Expertise: Craft and optimize complex T‑SQL queries to support efficient data processing and analytical workloads.

Required Qualifications

• 5+ years of hands‑on expertise with Azure Databricks, Python, PySpark, and Delta Lake.
• 5+ years of proven experience with Azure Data Factory for orchestrating and monitoring pipelines.
• Strong SQL Server / T‑SQL experience with a focus on query optimization, indexing strategies, and coding best practices.
• Demonstrated experience in SSIS package design, deployment, and performance tuning.
• Hands‑on knowledge of Unity Catalog for governance.
• Experience with Git (Azure DevOps Repos) and CI / CD practices in data engineering projects.

Nice to Have

• Exposure to Change Data Capture (CDC), Change Data Feed (CDF), and Temporal Tables.
• Experience with Microsoft Purview, Power BI, and Azure‑native integrations.
• Familiarity with Profisee Master Data Management (MDM).
• Working in Agile / Scrum environments.

Preferred Qualifications

• Microsoft Certified: Azure Data Engineer Associate (DP‑203)
• Microsoft Certified: Azure Solutions Architect Expert or equivalent advanced Azure certification
• Databricks Certified Data Engineer Associate or Professional
• Additional Microsoft SQL Server or Azure certifications demonstrating advanced database and cloud expertise