Senior Databricks & Governance Engineer

Papigen; India; Presencial; INR 3.000.000 - 5.500.000; Jornada completa

Descripción de la vacante

Papigen in India is seeking a Senior Databricks & Governance Engineer to build scalable data pipelines, govern data assets, and prepare unstructured data for Generative AI in a modern Lakehouse environment.

You will work hands-on with Azure Databricks, PySpark, Python, SQL, and Delta Lake; implement Unity Catalog governance; integrate Collibra via REST APIs; and design Bronze-Silver-Gold data models to support RAG, embeddings, and vector search.

Formación

• 5+ years of data engineering experience
• Hands-on Azure Databricks experience
• Proficiency with PySpark, Python, SQL and Delta Lake
• Experience with Unity Catalog governance and Collibra REST API integrations

Responsabilidades

• Develop and maintain batch and streaming ETL/ELT pipelines using Databricks, PySpark, Python, SQL, and Delta Lake
• Orchestrate data workflows using Azure Data Factory (ADF)
• Extract and process unstructured data from PDFs, images, logs, and text files
• Prepare data for RAG, LLM applications, embeddings, and vector search
• Implement data access controls, security, and lineage using Unity Catalog
• Integrate Databricks metadata with Collibra using REST APIs
• Design and maintain Bronze, Silver, and Gold layers using Medallion Architecture
• Develop dimensional and Lakehouse data models
• Optimize performance using Z-Ordering, Liquid Clustering, and Delta Lake optimization techniques
• Ensure data quality, governance, metadata consistency, and pipeline reliability

Conocimientos

Databricks

PySpark

Python

SQL

Delta Lake

Azure Data Factory

Unity Catalog

Collibra API

GenAI prep

RAG

Embeddings

Vector search

Medallion Architecture

Data governance

Data modeling

Herramientas

Azure Databricks

Collibra

Delta Lake

REST APIs

Descripción del empleo

Role Overview

We are looking for a Senior Databricks & Governance Engineer with 5+ years of Data Engineering experience to develop scalable data pipelines, implement enterprise data governance, and prepare unstructured data for Generative AI applications in a modern Lakehouse environment.

Key Responsibilities

• Develop and maintain batch and streaming ETL/ELT pipelines using Databricks, PySpark, Python, SQL, and Delta Lake.
• Orchestrate data workflows using Azure Data Factory (ADF).
• Extract and process unstructured data from PDFs, images, logs, and text files.
• Prepare data for RAG, LLM applications, embeddings, and vector search.
• Implement data access controls, security, and lineage using Unity Catalog.
• Integrate Databricks metadata with Collibra using REST APIs.
• Design and maintain Bronze, Silver, and Gold layers using Medallion Architecture.
• Develop dimensional and Lakehouse data models.
• Optimize performance using Z-Ordering, Liquid Clustering, and Delta Lake optimization techniques.
• Ensure data quality, governance, metadata consistency, and pipeline reliability.

Mandatory Skills

• 5+ years of Data Engineering experience
• Strong hands-on experience with Azure Databricks
• PySpark, Python, SQL, and Delta Lake
• Azure Data Factory (ADF)
• Unity Catalog — governance, access controls, and lineage
• Collibra REST API integrations — metadata extraction and synchronization
• GenAI / LLM data preparation, including unstructured data processing
• RAG, embeddings, and vector search
• Lakehouse and Medallion Architecture
• Dimensional data modeling and performance optimization

Skills: pyspark,azure data factory,collibra,genai,delta lake,unity catalog,sql,python,azure databricks