Data Solution Architect

Pyramid Consulting, Inc; México; On-site; MXN 900,000 - 1,300,000; Full time

Job summary

Pyramid Consulting, Inc is seeking a skilled Data Engineer with strong Databricks and Spark expertise to design end-to-end lakehouse architectures across Bronze, Silver, and Gold layers. You will optimize pipelines, implement Delta Lake features, and enforce governance using Unity Catalog.

The role requires advanced SQL and programming in Python or Scala, with a focus on production readiness and cost-efficient cloud deployments across AWS/Azure/GCP environments.

Qualifications

• Lakehouse & Medallion Architecture: Expertise in end-to-end data architectures (Bronze, Silver, Gold) for production-ready pipelines.
• Databricks & Spark Internals: Deep understanding of distributed computing; troubleshoot and tune large Spark jobs with caching, partitioning, and broadcast joins.
• Delta Lake: ACID transactions, schema enforcement, time travel, and optimization like Z-ordering.
• Unity Catalog & Data Governance: Design governance models with RBAC, row/column security, and data lineage.
• Cloud Infrastructure: Knowledge of AWS/Azure/GCP ecosystems, ADLS/Entra for Azure, S3/IAM for AWS, and VNet/IAM setups.
• Coding Proficiency: Advanced SQL and fluency in Python or Scala.
• Cost Optimization & Performance Tuning: Monitor DBUs and right-size clusters to avoid cloud bill shock.

Skills

Advanced SQL

Python

Scala

Spark proficiency

Tools

Databricks

Spark

Delta Lake

Unity Catalog

ADLS/Entra (Azure)

S3/IAM (AWS)

Job description

"""Overall Stack: Databricks Lakehouse Platform, Apache Spark, Delta Lake, Unity Catalog, and modern cloud data architecture

Must Have

Lakehouse & Medallion Architecture: Expertise in designing end-to-end data architectures (Bronze, Silver, Gold layers) for reliable, production-ready pipelines

Databricks & Spark Internals: Deep understanding of distributed computing. They must know how to troubleshoot and tune large-scale Spark jobs using caching, partitioning, and broadcast joins

Delta Lake: Must understand ACID transactions, schema enforcement, time travel, and optimization operations like Z-ordering

Unity Catalog & Data Governance: Proven ability to design unified governance models for data and AI assets, including role-based access control (RBAC), row/column-level security, and data lineage

Cloud Infrastructure (AWS, Azure, or GCP): Strong grasp of the native cloud ecosystem they work in (e.g., ADLS/Entra for Azure, S3/IAM for AWS), including Virtual Network (VNet) setups and IAM roles

Coding Proficiency: Advanced SQL skills and fluency in Python or Scala

Cost Optimization & Performance Tuning: Ability to monitor DBUs (Databricks Units), right-size serverless and multi-node clusters, and implement best practices for avoiding cloud bill shock

Nice to Have

Databricks Certifications: Candidates holding valid Databricks Certified Data Architect or Databricks Certified Data Engineer Professional badges generally have a proven, up-to-date baseline of the platform's features

Generative AI & MLflow Integration: Experience building, deploying, and monitoring GenAI applications and ML models using Databricks Model Serving, Vector Search, and the Mosaic AI suite

CI/CD & DevOps Practices: Experience automating Databricks workflows using Git (Databricks Repos) and orchestration tools like dbt, Azure Data Factory, or Apache Airflow

Streaming Data: Familiarity with Databricks Structured Streaming and Auto Loader for real-time data ingestion and processing

Data Warehousing & BI: Understanding of Databricks SQL, Serverless Warehouses, and integration with downstream BI tools like Power BI