Senior Data Engineer

Trigent Software; Thane; Presencial; INR 2.800.000 - 4.200.000; Jornada completa

Descripción de la vacante

Trigent Software is seeking a Data Engineer with 7–10 years of experience focused on Big Data platforms. The role requires hands-on expertise in Azure Databricks and Delta Lake to design and operate robust lakehouse architectures.

You will implement Medallion Architecture in production, develop with PySpark/Spark SQL, and own data orchestration via Azure Data Factory. Additional requirements include ADLS Gen2, Unity Catalog, and strong CI/CD practices using Azure DevOps, Git, and GitHub Actions.

Formación

• 7 - 10 years of overall experience in Data Engineering with a focus on Big Data platforms.
• Hands-on experience with Azure Databricks is mandatory.
• Extensive experience building Data Lake / Lakehouse architectures using Delta Lake.
• Proven expertise implementing Medallion Architecture (Bronze/Silver/Gold) in production environments.
• Strong programming skills in PySpark, Spark SQL, and Python.
• Solid understanding of distributed computing concepts and Spark internals.
• Hands-on experience with Azure Data Factory for orchestration and data ingestion.
• Experience with ADLS Gen2 and related Azure data services (Event Hubs, Key Vault).
• Experience with Unity Catalog.
• SQL skills and data modeling (dimensional modeling, star/snowflake schemas) preferred.
• Experience with CI/CD practices for data pipelines (Azure DevOps, Git, GitHub Actions).
• Familiarity with streaming data pipelines (Structured Streaming, Auto Loader, Kafka/Event Hubs) is a plus.
• Understanding of data quality frameworks, monitoring, and observability tools.

Herramientas

Azure Databricks

Delta Lake

Medallion Architecture

PySpark

Spark SQL

Python

Spark Internals

Azure Data Factory

Azure Data Lake Storage Gen2

Unity Catalog

Azure DevOps

Git

GitHub Actions

Structured Streaming

Kafka/Event Hubs

Observability Tools

Descripción del empleo

• 7 - 10 years of overall experience in Data Engineering, with a strong focus on Big Data platforms.
  • Mandatory hands‑on experience with Azure Databricks - this is a core requirement, not optional.
  • Strong experience building and managing Data Lake / Lakehouse architectures using Delta Lake.
  • Proven expertise implementing Medallion Architecture (Bronze/Silver/Gold) in production environments.
  • Strong programming skills in PySpark, Spark SQL, and Python.
  • Solid understanding of distributed computing concepts and Spark internals (partitioning, caching, shuffling, optimization techniques).
  • Hands‑on experience with Azure Data Factory (ADF) for orchestration and data ingestion.
• Experience with Azure Data Lake Storage Gen2 (ADLS) and related Azure data services (Event Hubs, Key Vault etc.).
• Experience with Unity Catalog
• SQL skills and experience with data modeling (dimensional modeling, star/snowflake schemas)(Preferred)
• Experience with CI/CD practices for data pipelines (Azure DevOps, Git, GitHub Actions).
• Familiarity with streaming data pipelines (Structured Streaming, Auto Loader, Kafka/Event Hubs) is a plus.
• Understanding of data quality frameworks, monitoring, and observability tools.