Databrick Architect

Pyramid Consulting, Inc; Estado de México; On-site; MXN 900,000 - 1,700,000; Full time

Job summary

Pyramid Consulting, Inc. seeks a seasoned Databricks Architect to design and implement enterprise Databricks Lakehouse platforms. You will lead end-to-end data architectures for batch and streaming analytics, aligning cloud-native strategies with business goals.

Role requires hands-on Spark/PySpark, data governance and security expertise, and strong stakeholder leadership across BFSI contexts. You will mentor engineers and drive standardized best practices across cloud platforms.

Qualifications

• 12+ years of IT experience
• 5+ years of Databricks architecture experience
• Hands-on Spark/PySpark experience
• Strong lakehouse architecture expertise
• Data governance & security expertise
• Stakeholder management & leadership experience

Responsibilities

• Design and implement enterprise-scale Databricks Lakehouse platforms.
• Define end-to-end data architecture for batch, streaming, and real-time analytics solutions.
• Develop cloud-native data platform strategies aligned with business objectives.
• Establish architectural standards, governance frameworks, and best practices.
• Lead modernization initiatives involving legacy data warehouse migration to Databricks.
• Design scalable ETL/ELT pipelines using Databricks, Apache Spark, and Client Lake.
• Build medallion architecture (Bronze, Silver, Gold) for enterprise data platforms.
• Integrate data from multiple sources including databases, APIs, SaaS platforms, and streaming systems.
• Implement data quality, lineage, observability, and monitoring frameworks.
• Optimize data ingestion and transformation processes for performance and scalability.
• Architect Databricks solutions on Azure, AWS, or Client Cloud Platform.
• Design secure networking, storage, identity management, and access controls.
• Optimize platform usage and cloud costs through governance and monitoring.
• Lead Databricks workspace design and multi-environment deployment strategies.

Skills

Databricks architecture

Spark/PySpark

Leadership

Stakeholder management

Data governance

Security

Education

Bachelor's degree in CS/Engineering/IS

Tools

Unity Catalog

Apache Ranger

Job description

Former Databricks Employee (Non-Negotiable Requirement)

12+ Years of IT Experience

5+ Years of Databricks Architecture Experience

Strong Lakehouse Architecture Expertise

Hands-on Spark/PySpark Experience

Data Governance & Security Expertise

Stakeholder Management and Leadership Experience

Preferred

BFSI Domain Experience

Databricks Technologies

• Unity Catalog
• Data Warehousing

OR

DevOps & Automation

Data Governance & Security

• Unity Catalog
• Apache Ranger
• Data Quality Frameworks

________________________________________

Domain Experience (Preferred)

• Banking & Financial Services (BFSI)
• Regulatory Reporting and Compliance

________________________________________

Education

• Bachelor's Degree in Computer Science, Engineering, Information Systems, or related field.

________________________________________

Certifications (Preferred)

• Databricks Certified Data Engineer Professional
• Azure Solutions Architect Expert

Roles and Responsibilities:

Architecture & Strategy

• Design and implement enterprise-scale Databricks Lakehouse platforms.
• Define end-to-end data architecture for batch, streaming, and real-time analytics solutions.
• Develop cloud-native data platform strategies aligned with business objectives.
• Establish architectural standards, governance frameworks, and best practices.
• Lead modernization initiatives involving legacy data warehouse migration to Databricks.
• Design scalable ETL/ELT pipelines using Databricks, Apache Spark, and Client Lake.
• Build medallion architecture (Bronze, Silver, Gold layers) for enterprise data platforms.
• Integrate data from multiple sources including databases, APIs, SaaS platforms, and streaming systems.
• Implement data quality, lineage, observability, and monitoring frameworks.
• Optimize data ingestion and transformation processes for performance and scalability.
• Architect Databricks solutions on Azure, AWS, or Client Cloud Platform.
• Design secure networking, storage, identity management, and access controls.
• Optimize platform usage and cloud costs through governance and monitoring.
• Lead Databricks workspace design and multi-environment deployment strategies.

AI, ML & Advanced Analytics

• Architect Databricks solutions supporting Machine Learning and Generative AI workloads.
• Enable enterprise AI capabilities leveraging Databricks Mosaic AI and GenAI frameworks.
• Collaborate with Data Scientists and AI teams to operationalize ML models.
• Establish MLOps best practices and governance standards.

Security & Governance

• Implement Unity Catalog for enterprise-wide governance.
• Define data security controls including RBAC, ABAC, encryption, and access management.
• Ensure compliance with industry standards including SOX, PCI-DSS, HIPAA, GDPR, and BFSI regulatory requirements.
• Establish data governance, metadata management, and audit frameworks.

Leadership & Stakeholder Management

• Act as the primary Databricks subject matter expert.
• Engage with executive stakeholders to define data transformation roadmaps.
• Conduct architecture reviews and provide technical leadership.
• Mentor data engineers, platform engineers, and architects.
• Collaborate with business, infrastructure, security, and application teams.