Senior Databricks Lakehouse Architect for AI & Governance

Pyramid Consulting, Inc; Estado de México; On-site; MXN 900,000 - 1,700,000; Full time

Job summary

Pyramid Consulting, Inc. seeks a seasoned Databricks Architect to design and implement enterprise Databricks Lakehouse platforms. You will lead end-to-end data architectures for batch and streaming analytics, aligning cloud-native strategies with business goals.

Role requires hands-on Spark/PySpark, data governance and security expertise, and strong stakeholder leadership across BFSI contexts. You will mentor engineers and drive standardized best practices across cloud platforms.

Qualifications

• 12+ years of IT experience
• 5+ years of Databricks architecture experience
• Hands-on Spark/PySpark experience
• Strong lakehouse architecture expertise
• Data governance & security expertise
• Stakeholder management & leadership experience

Responsibilities

• Design and implement enterprise-scale Databricks Lakehouse platforms.
• Define end-to-end data architecture for batch, streaming, and real-time analytics solutions.
• Develop cloud-native data platform strategies aligned with business objectives.
• Establish architectural standards, governance frameworks, and best practices.
• Lead modernization initiatives involving legacy data warehouse migration to Databricks.
• Design scalable ETL/ELT pipelines using Databricks, Apache Spark, and Client Lake.
• Build medallion architecture (Bronze, Silver, Gold) for enterprise data platforms.
• Integrate data from multiple sources including databases, APIs, SaaS platforms, and streaming systems.
• Implement data quality, lineage, observability, and monitoring frameworks.
• Optimize data ingestion and transformation processes for performance and scalability.
• Architect Databricks solutions on Azure, AWS, or Client Cloud Platform.
• Design secure networking, storage, identity management, and access controls.
• Optimize platform usage and cloud costs through governance and monitoring.
• Lead Databricks workspace design and multi-environment deployment strategies.

Skills

Databricks architecture

Spark/PySpark

Leadership

Stakeholder management

Data governance

Security

Education

Bachelor's degree in CS/Engineering/IS

Tools

Unity Catalog

Apache Ranger

Job description

Pyramid Consulting, Inc. seeks a seasoned Databricks Architect to design and implement enterprise Databricks Lakehouse platforms. You will lead end-to-end data architectures for batch and streaming analytics, aligning cloud-native strategies with business goals.

Role requires hands-on Spark/PySpark, data governance and security expertise, and strong stakeholder leadership across BFSI contexts. You will mentor engineers and drive standardized best practices across cloud platforms.