Senior Data Engineer - Brazil

Papigen; Brasília; Presencial; BRL 260 000 - 420 000; Tempo integral

Resumo da oferta

Papigen in Brasília seeks a Senior Data Engineer/Databricks Engineer with 8+ years of IT experience to design, develop, optimize, and support enterprise data pipelines and analytical datasets. You will work with Azure Databricks, ADLS, ADF, PostgreSQL, and Azure DevOps to deliver reliable data ingestion, transformation, reconciliation, and reporting-ready data solutions.

Key responsibilities include building ETL/ELT pipelines, developing PySpark notebooks, implementing Medallion Architecture,

Qualificações

• 8+ years of IT experience with strong data engineering background.
• Proficiency in Azure Databricks notebooks, PySpark, SQL and Python.
• Experience with ADLS, ADF, and PostgreSQL for data ingestion and modeling.
• Knowledge of Medallion Architecture (Bronze/Silver/Gold).
• Solid data quality, validation, reconciliation, and exception handling.
• Ability to optimize pipelines and troubleshoot production issues.

Responsabilidades

• Design, develop, and maintain ETL/ELT pipelines for ingestion and transformation.
• Build Databricks notebooks and reusable data engineering components using PySpark and SQL.
• Implement Medallion Architecture across Bronze, Silver, Gold layers.
• Integrate diverse data sources including financial, accounting, or enterprise data.
• Develop analytical datasets and data models for reporting and analytics.
• Implement data-quality checks, monitoring, and exception handling.
• Tune Databricks workloads and PostgreSQL data processing for performance.
• Monitor scheduled loads, troubleshoot failures, and provide runbooks.
• Support data lineage, metadata, and source-to-target mapping; participate in CI/CD and release processes.
• Collaborate with architects, BI developers, QA, business analysts, and DevOps.

Conhecimentos

8+ years IT experience

Azure Databricks

ADLS

PySpark

Python

SQL

ETL/ELT

Medallion architecture

PostgreSQL

Data quality

Performance tuning

Production support

Data pipeline monitoring

Ferramentas

Azure Data Lake Storage (ADLS)

Azure Data Factory

Azure DevOps

Git

Delta Lake

Databricks notebooks

SQL tooling

Descrição da oferta de emprego

Role Overview

We are looking for a Senior Data Engineer / Databricks Engineer with 8+ years of experience to design, develop, optimize, and support enterprise data pipelines and analytical datasets.

The candidate will work with Azure Databricks, Azure Data Lake, Azure Data Factory, PostgreSQL, and Azure DevOps to deliver reliable data ingestion, transformation, reconciliation, and reporting-ready data solutions.

Key Responsibilities

• Design, develop, and maintain ETL/ELT pipelines for data ingestion, transformation, and integration.
• Develop Databricks notebooks and reusable data engineering components using PySpark and SQL.
• Implement Medallion Architecture (Bronze, Silver, and Gold layers).
• Integrate operational, financial, accounting, portfolio, risk, and enterprise data sources.
• Develop curated analytical datasets and data models for reporting and analytics.
• Implement data-quality checks, validation rules, reconciliation controls, and exception handling.
• Optimize Databricks workloads, SQL queries, and PostgreSQL-related data processing.
• Monitor scheduled data loads and troubleshoot pipeline failures and production incidents.
• Support data lineage, metadata, source-to-target mapping, and technical documentation.
• Implement CI/CD, source control, deployment validation, and release-management processes.
• Collaborate with architects, BI developers, QA analysts, business analysts, and DevOps teams.
• Maintain pipeline documentation, operational runbooks, and monitoring reports.

Mandatory Skills & Experience

• 8+ years of overall IT experience, with strong relevant Data Engineering experience.
• Azure Databricks - notebooks, transformations, and pipeline development.
• Azure Data Lake Storage (ADLS).
• PySpark, Python, and SQL.
• ETL/ELT pipeline development and data integration.
• Medallion Architecture - Bronze, Silver, and Gold layers.
• Strong SQL and PostgreSQL skills.
• Data quality, reconciliation, validation, and exception handling.
• Performance tuning and production troubleshooting.
• Data pipeline monitoring and scheduled job support.

Preferred Skills

• Azure Data Factory (ADF).
• Azure DevOps, Git, CI/CD pipelines, and automated deployments.
• Data lineage, metadata management, and source-to-target mapping.
• Delta Lake and lakehouse data modeling.
• Batch processing, job recovery, and operational automation.
• Financial services, accounting, portfolio, risk, or reinsurance data.
• Release management, root-cause analysis, and production support.