Databricks Azure SME

Pyramid Consulting, Inc; Ciudad de México; On-site; MXN 900,000 - 1,200,000; Full time

Job summary

Pyramid Consulting, Inc. seeks a Databricks SME on Azure to design and implement scalable Databricks environments on the Azure platform. You will work with data engineering, analytics, and IT teams to deploy, optimize, and securely manage Databricks solutions across cloud resources.

The role requires deep Databricks expertise and extensive experience with Azure cloud services, including DevOps practices for CI/CD and automation. Collaboration across teams is essential.

Qualifications

• Deep expertise in Databricks and Azure cloud services.
• Experience designing, implementing, and managing Databricks environments on Azure.
• Ability to work with data engineering, analytics, and IT teams to deploy and manage Databricks solutions.

Responsibilities

• Architect and deploy scalable Databricks solutions on Azure.
• Develop data pipelines, ETL processes, and ML workflows using Databricks and Azure services.
• Monitor Databricks performance and optimize configurations for cost and efficiency.
• Ensure Databricks environments comply with security standards and regulatory requirements.
• Perform maintenance tasks, updates, patches, and backups; provide advanced support.
• Collaborate with data engineers, scientists, and analysts to optimize data workflows.
• Deliver training on Databricks best practices and document architectures and procedures.

Skills

Databricks architecture

Spark & Delta Lake

Azure services

Python/SQL/Scala

Azure DevOps

Data Factory

Power BI

Cloud cost optimization

Tools

Azure DevOps

Jenkins

GitHub Actions

Nexus

JIRA

Job description

• The Databricks SME on Azure is responsible for the design, implementation, and management of Databricks environments on the Azure platform.
• This role requires deep expertise in Databricks and extensive experience with Azure cloud services.
• The SME will work closely with data engineering, analytics, and IT teams to ensure the efficient deployment and management of Databricks solutions, driving data-driven decision-making across the organization.
• The SME will collaborate with cross-functional teams along with other HR Integration SMEs and architects to provide technical expertise and ensure the efficient deployment and management of cloud resources.

Core Responsibilities:

Design and Implementation:

• Architect and deploy scalable Databricks solutions on Azure.
• Develop and implement data pipelines, ETL processes, and machine learning workflows using Databricks and Azure services.

Performance and Optimization:

• Monitor Databricks performance and optimize configurations for efficiency and cost-effectiveness.
• Implement best practices for data storage, processing, and analysis to ensure optimal performance.

Security and Compliance:

• Ensure Databricks environments comply with security standards and regulatory requirements.
• Implement and manage data encryption, access controls, and other security measures.

Maintenance and Support:

• Perform regular maintenance tasks, including updates, patches, and backups.
• Provide advanced technical support for Databricks-related issues and act as the escalation point for complex problems.

Collaboration and Consultation:

• Collaborate with data engineers, data scientists, and analysts to design and optimize data workflows.
• Work with IT and DevOps teams to integrate Databricks with other Azure services and tools.

Training and Documentation:

• Develop and deliver training sessions for internal teams on Databricks best practices and new features.
• Document architectures, configurations, and procedures for knowledge sharing and future reference.
• Strong analytical and problem-solving skills.
• Strong technical documentation capabilities.
• Agile framework and delivery lifecycle management.
• Team based development capabilities and ability to manage personal deliverables.
• Experience with multi-cloud environments and hybrid cloud strategies.
• Familiarity with DevOps practices and tools such as Azure DevOps, Jenkins, or GitHub Actions.
• Databricks Certified Associate Developer or equivalent certification.
• Relevant Azure certifications, such as Azure Data Engineer Associate or Azure Solutions Architect

Skills:

• In-depth knowledge of Databricks architecture, Spark, and Delta Lake.
• Proficiency in managing and optimizing Databricks clusters and jobs.
• Experience with deploying and managing Databricks on Azure, including integration with Azure Data Lake, Azure SQL Database, and other Azure services.
• Strong programming skills in languages such as Python, SQL, and Scala.
• Familiarity with data warehousing concepts and ETL processes.
• Strong understanding of Azure infrastructure and services, such as Azure Virtual Machines, Azure Storage, and Azure Networking.
• Azure Database and data platform
• DevOps Tooling: Jenkins, Nexus, GitHub, JIRA
• Database Services, including Azure SQL and Cosmos DB
• DevOps Practices with Azure DevOps
• Monitoring, Diagnostics, and Analytics with Azure Monitor

Additional Skills:

• Azure Data Factory and Data Pipeline Orchestration
• Azure Synapse Analytics and Data Warehousing
• Data Modeling and Database Design
• Power BI and Data Visualization
• Machine Learning and AI Integration with Azure ML