Data Solution Architect

Pyramid Consulting, Inc; México; Sur place; MXN 700 000 - 1 100 000; Plein temps

Résumé du poste

Pyramid Consulting, Inc. seeks a senior data engineer focused on Databricks Lakehouse platforms in Mexico. You will design end-to-end data architectures across Bronze, Silver, and Gold layers, optimize large Spark jobs, and implement robust data governance using Unity Catalog and RBAC.

You should master SQL with Python or Scala, tune performance, and handle cloud infrastructure like ADLS/S3 with IAM roles. Exposure to GenAI and CI/CD is a plus.

Qualifications

• End-to-end data architectures (Bronze/Silver/Gold) design.
• Deep understanding of distributed Spark, tuning and caching.
• ACID, schema enforcement, time travel in Client Lake, data optimization.
• Governance design: RBAC, row/column security, data lineage.
• Cloud infra familiarity: AWS/Azure/GCP with IAM and VNet setups.
• Advanced SQL and fluency in Python or Scala.

Connaissances

Lakehouse Architecture

Databricks

Spark Internals

Unity Catalog

Cloud Platforms

SQL

Python/Scala

Outils

dbt

Airflow

Git (Databricks Repos)

Azure Data Factory

Description du poste

\"\"\"Overall Stack: Databricks Lakehouse Platform, Apache Spark, Client Lake, Unity Catalog, and modern cloud data architecture

Must Have

• Lakehouse & Medallion Architecture: Expertise in designing end-to-end data architectures (Bronze, Silver, Gold layers) for reliable, production-ready pipelines
• Databricks & Spark Internals: Deep understanding of distributed computing. They must know how to troubleshoot and tune large-scale Spark jobs using caching, partitioning, and broadcast joins
• Client Lake: Must understand ACID transactions, schema enforcement, time travel, and optimization operations like Z-ordering
• Unity Catalog & Data Governance: Proven ability to design unified governance models for data and AI assets, including role-based access control (RBAC), row/column-level security, and data lineage
• Cloud Infrastructure (AWS, Azure, or GCP): Strong grasp of the native cloud ecosystem they work in (e.g., ADLS/Entra for Azure, S3/IAM for AWS), including Virtual Network (VNet) setups and IAM roles
• Coding Proficiency: Advanced SQL skills and fluency in Python or Scala
• Cost Optimization & Performance Tuning: Ability to monitor DBUs (Databricks Units), right-size serverless and multi-node clusters, and implement best practices for avoiding cloud bill shock

Nice to Have

• Databricks Certifications: Candidates holding valid Databricks Certified Data Architect or Databricks Certified Data Engineer Professional badges generally have a proven, up-to-date baseline of the platform’s features
• Generative AI & MLflow Integration: Experience building, deploying, and monitoring GenAI applications and ML models using Databricks Model Serving, Vector Search, and the Mosaic AI suite
• CI/CD & DevOps Practices: Experience automating Databricks workflows using Git (Databricks Repos) and orchestration tools like dbt, Azure Data Factory, or Apache Airflow
• Streaming Data: Familiarity with Databricks Structured Streaming and Auto Loader for real-time data ingestion and processing
• Data Warehousing & BI: Understanding of Databricks SQL, Serverless Warehouses, and integration with downstream BI tools like Power BI