Databricks + Azure Data Engineer

Casado; Brazil; Remote; BRL 180,000 - 300,000; Full time

Benefits offered by this job

Databricks platform access

Job summary

Casado is hiring a senior Data Engineer to design, build, and optimize data pipelines on Azure Databricks and PySpark. The role focuses on reliable ingestion, transformation, and governance across ADLS Gen2 and Delta Lake in a cloud data lakehouse.

You will implement Medallion architecture, enhance CI/CD practices with Azure DevOps, and collaborate with architects, analysts, and BI teams in an Agile setting to meet SLAs.

Qualifications

• 7–10+ years in Data Engineering or Azure cloud data development.
• Hands-on with Azure Databricks, PySpark, Spark SQL, and Delta Lake.
• Proven experience building scalable ETL/ELT pipelines with ADF.
• Solid grasp of ADLS Gen2, Lakehouse architecture, and distributed data processing.
• CI/CD with Azure DevOps, Git, YAML pipelines, and IaC (Terraform/ARM/Bicep).
• Strong SQL skills with performance-tuning expertise.
• Experience in Agile/Scrum environments.

Responsibilities

• Design, develop, and maintain data pipelines using Azure Databricks and PySpark.
• Build and orchestrate ETL/ELT workflows with ADF.
• Develop ingestion & transformation pipelines using ADLS Gen2, Delta Lake, and Spark SQL.
• Implement Medallion architecture (Bronze/Silver/Gold), incremental loads, CDC, MERGE, and SCD Type 1/2.
• Optimize Spark jobs via partitioning, caching, file-size optimization, and performance tuning.
• Configure and maintain Databricks Workflows/Jobs, clusters, and workspaces.
• Build data quality checks, validation, error handling, monitoring, and reconciliation.
• Leverage Unity Catalog for governance, lineage, RBAC, and secure data access.
• Integrate Databricks with Azure services (ADLS, Azure SQL, APIs, JDBC).
• Implement CI/CD with Azure DevOps & Git; automate IaC with Terraform or ARM/Bicep.
• Troubleshoot production pipelines, run root-cause analysis, and ensure SLA compliance.
• Collaborate with architects, analysts, and BI teams in an Agile environment.

Skills

Azure Databricks

PySpark

Spark SQL

Delta Lake

ETL/ELT pipelines

Azure Data Factory

ADLS Gen2

CI/CD

Terraform

SQL performance tuning

Agile/Scrum

Tools

Azure DevOps

Git

YAML pipelines

Terraform

ARM/Bicep

Databricks

Job description

We're Hiring: Databricks + Azure Data Engineer

Location: LATAM based (Remote)

What You'll Do

• Design, develop, and maintain data pipelines using Azure Databricks and PySpark
• Build and orchestrate ETL/ELT workflows with Azure Data Factory (ADF)
• Develop ingestion & transformation pipelines using ADLS Gen2, Delta Lake, and Spark SQL
• Implement Medallion architecture (Bronze/Silver/Gold), incremental loads, CDC, MERGE, and SCD Type 1/2
• Optimize Spark jobs via partitioning, caching, file-size optimization, and performance tuning
• Configure and maintain Databricks Workflows/Jobs, clusters, and workspaces
• Build data quality checks, validation, error handling, monitoring, and reconciliation
• Leverage Unity Catalog for governance, lineage, RBAC, and secure data access
• Integrate Databricks with Azure services (ADLS, Azure SQL, APIs, JDBC)
• Implement CI/CD with Azure DevOps & Git; automate IaC with Terraform or ARM/Bicep
• Troubleshoot production pipelines, run root-cause analysis, and ensure SLA compliance
• Collaborate with architects, analysts, and BI teams in an Agile environment

What You Bring

• 7–10+ years in Data Engineering or Azure cloud data development
• Strong hands-on with Azure Databricks, PySpark, Spark SQL, and Delta Lake
• Proven experience building scalable ETL/ELT pipelines with ADF
• Solid grasp of ADLS Gen2, Lakehouse architecture, and distributed data processing
• CI/CD (Azure DevOps), Git, YAML pipelines, and IaC (Terraform/ARM/Bicep)
• Strong SQL skills with performance-tuning expertise
• Experience in Agile/Scrum environments

Nice to Have

• Databricks platform administration (clusters, jobs, workspace config)
• Unity Catalog, data governance, lineage & RBAC know-how
• Databricks integration with APIs, JDBC sources & external systems
• Familiarity with Datadog/Grafana and cost optimization