Senior Databricks & Governance Engineer

Papigen; Brasil; Presencial; BRL 240.000 - 360.000; Jornada completa

Descripción de la vacante

Papigen is seeking a Senior Databricks & Governance Engineer to build scalable data pipelines, enforce governance, and prepare unstructured data for Generative AI in a modern Lakehouse. You will implement end-to-end ETL/ELT processes, manage data quality, and optimize metadata and security using Unity Catalog and Collibra integrations.

Ideal candidates have 5+ years in data engineering, hands-on Azure Databricks and Delta Lake skills, and experience with GenAI data preparation, embeddings, and

Formación

• 5+ years of Data Engineering experience.
• Hands-on Azure Databricks expertise.
• Experience with unstructured data and GenAI preparation.
• Experience with Delta Lake and Medallion architecture.

Responsabilidades

• Develop and maintain batch and streaming ETL/ELT pipelines using Databricks, PySpark, Python, SQL, and Delta Lake.
• Orchestrate data workflows using Azure Data Factory (ADF).
• Extract and process unstructured data from PDFs, images, logs, and text files.
• Prepare data for RAG, LLM applications, embeddings, and vector search.
• Implement data access controls, security, and lineage using Unity Catalog.
• Integrate Databricks metadata with Collibra using REST APIs.
• Design and maintain Bronze, Silver, and Gold layers using Medallion Architecture.
• Develop dimensional and Lakehouse data models.
• Optimize performance using Z-Ordering, Liquid Clustering, and Delta Lake optimization techniques.
• Ensure data quality, governance, metadata consistency, and pipeline reliability.

Conocimientos

Azure Databricks

PySpark

Python

SQL

Delta Lake

Azure Data Factory

Unity Catalog

Collibra REST API

GenAI prep

RAG

Embeddings

Vector search

Medallion Architecture

Dimensional modeling

Performance tuning

Descripción del empleo

Role Overview

We are looking for a Senior Databricks & Governance Engineer with 5+ years of Data Engineering experience to develop scalable data pipelines, implement enterprise data governance, and prepare unstructured data for Generative AI applications in a modern Lakehouse environment.

Key Responsibilities

• Develop and maintain batch and streaming ETL/ELT pipelines using Databricks, PySpark, Python, SQL, and Delta Lake.
• Orchestrate data workflows using Azure Data Factory (ADF).
• Extract and process unstructured data from PDFs, images, logs, and text files.
• Prepare data for RAG, LLM applications, embeddings, and vector search.
• Implement data access controls, security, and lineage using Unity Catalog.
• Integrate Databricks metadata with Collibra using REST APIs.
• Design and maintain Bronze, Silver, and Gold layers using Medallion Architecture.
• Develop dimensional and Lakehouse data models.
• Optimize performance using Z-Ordering, Liquid Clustering, and Delta Lake optimization techniques.
• Ensure data quality, governance, metadata consistency, and pipeline reliability.

Mandatory Skills

• 5+ years of Data Engineering experience
• Strong hands-on experience with Azure Databricks
• PySpark, Python, SQL, and Delta Lake
• Azure Data Factory (ADF)
• Unity Catalog — governance, access controls, and lineage
• Collibra REST API integrations — metadata extraction and synchronization
• GenAI / LLM data preparation, including unstructured data processing
• RAG, embeddings, and vector search
• Lakehouse and Medallion Architecture
• Dimensional data modeling and performance optimization

Skills: collibra,pyspark,delta lake,genai,azure data factory,unity catalog,sql,python,azure databricks