Databricks Lakehouse Data Solution Architect

Pyramid Consulting, Inc; México; On-site; MXN 900,000 - 1,300,000; Full time

Job summary

Pyramid Consulting, Inc is seeking a skilled Data Engineer with strong Databricks and Spark expertise to design end-to-end lakehouse architectures across Bronze, Silver, and Gold layers. You will optimize pipelines, implement Delta Lake features, and enforce governance using Unity Catalog.

The role requires advanced SQL and programming in Python or Scala, with a focus on production readiness and cost-efficient cloud deployments across AWS/Azure/GCP environments.

Qualifications

• Lakehouse & Medallion Architecture: Expertise in end-to-end data architectures (Bronze, Silver, Gold) for production-ready pipelines.
• Databricks & Spark Internals: Deep understanding of distributed computing; troubleshoot and tune large Spark jobs with caching, partitioning, and broadcast joins.
• Delta Lake: ACID transactions, schema enforcement, time travel, and optimization like Z-ordering.
• Unity Catalog & Data Governance: Design governance models with RBAC, row/column security, and data lineage.
• Cloud Infrastructure: Knowledge of AWS/Azure/GCP ecosystems, ADLS/Entra for Azure, S3/IAM for AWS, and VNet/IAM setups.
• Coding Proficiency: Advanced SQL and fluency in Python or Scala.
• Cost Optimization & Performance Tuning: Monitor DBUs and right-size clusters to avoid cloud bill shock.

Skills

Advanced SQL

Python

Scala

Spark proficiency

Tools

Databricks

Spark

Delta Lake

Unity Catalog

ADLS/Entra (Azure)

S3/IAM (AWS)

Job description

Pyramid Consulting, Inc is seeking a skilled Data Engineer with strong Databricks and Spark expertise to design end-to-end lakehouse architectures across Bronze, Silver, and Gold layers. You will optimize pipelines, implement Delta Lake features, and enforce governance using Unity Catalog.

The role requires advanced SQL and programming in Python or Scala, with a focus on production readiness and cost-efficient cloud deployments across AWS/Azure/GCP environments.