Azure Databricks Architect

Pyramid Consulting, Inc; Miami (FL); On-site; USD 150,000 - 160,000; Full time

Job summary

Pyramid Consulting, Inc. is seeking an Azure Databricks Architect for a fulltime onsite role in Miami, FL. The candidate will own end-to-end data platform architecture, emphasize performance, governance, and cost efficiency, and collaborate with analysts and engineering teams.

Requires 12+ years in data engineering/architecture, hands-on Azure Databricks, Spark, Python/Scala, SQL, and Delta Lake, plus Azure DevOps/CI/CD experience. Certifications and financial services knowledge are a plus.

Qualifications

• 12+ years of experience in data engineering or architecture.
• Hands-on with Azure Databricks, Spark, Python/Scala, SQL, and Delta Lake.
• Design cloud data hubs, data lakes, and analytics stores.
• Experience with Azure DevOps, CI/CD, and automated deployments.
• Ability to implement solutions rather than only oversee.
• Strong communication and stakeholder collaboration.
• Experience in financial services or insurance.
• Knowledge of Event Hub, Cosmos DB, Azure Synapse, or similar services.
• Databricks or Microsoft Azure certifications.
• Experience with reusable frameworks, shell scripting, and production support.

Responsibilities

• Own the end-to-end architecture for Azure Databricks-based data platforms, including ingestion, transformation, storage, serving, and governance layers.
• Design reusable frameworks, ELT patterns, and data models that improve delivery speed, consistency, and maintainability.
• Build and optimize pipelines and Spark jobs directly, with a strong focus on performance, reliability, and cost efficiency.
• Architect solutions using Azure Databricks, Delta Lake, ADLS, Azure Data Factory, and related Azure platform services.
• Establish governance, security and code-quality standards for the platform.
• Partner closely with business analysts, functional teams, and engineering teams to translate requirements into working solutions.
• Guide troubleshooting, code reviews, and production support for complex data workloads.
• Drive modernization and cloud-native adoption across data and analytics capabilities

Skills

Azure Databricks

Spark

Python/Scala

SQL

Delta Lake

Azure DevOps

CI/CD

Communication

Job description

Immediate need for a talented Azure Databricks Architect. This is a Fulltime opportunity with long-term potential and is located in Miami, FL(Onsite).

Job ID: 26-22904

Pay Range: $150000 - $160000/annum. Employee benefits include, but are not limited to, health insurance (medical, dental, vision), 401(k) plan, and paid sick leave (depending on work location).

Key Responsibilities:

• Own the end-to-end architecture for Azure Databricks-based data platforms, including ingestion, transformation, storage, serving, and governance layers.
• Design reusable frameworks, ELT patterns, and data models that improve delivery speed, consistency, and maintainability.
• Build and optimize pipelines and Spark jobs directly, with a strong focus on performance, reliability, and cost efficiency.
• Architect solutions using Azure Databricks, Delta Lake, ADLS, Azure Data Factory, and related Azure platform services.
• Establish governance, security and code-quality standards for the platform.
• Partner closely with business analysts, functional teams, and engineering teams to translate requirements into working solutions.
• Guide troubleshooting, code reviews, and production support for complex data workloads.
• Drive modernization and cloud-native adoption across data and analytics capabilities

Key Requirements and Technology Experience:

• Key skills; Azure Databricks, Spark, Python/Scala, SQL, and Delta Lake, Azure DevOps, CI/CD.
• 12+ years of experience in data engineering, data architecture, or platform engineering.
• Strong hands-on experience with Azure Databricks, Spark, Python/Scala, SQL, and Delta Lake.
• Proven experience designing and implementing cloud data hubs, data lakes, and analytical data stores.
• Experience with Azure DevOps, CI/CD, and automated deployment practices.
• Ability to work directly on technical implementation, not only provide oversight.
• Strong communication and stakeholder collaboration skills.
• Experience in financial services or insurance.
• Knowledge of Event Hub, Cosmos DB, Azure Synapse, or other Azure analytics services.
• Databricks or Microsoft Azure certifications.
• Experience with reusable frameworks, shell scripting, and production support in large-scale environments.

Our client is a leading IT Industry, and we are currently interviewing to fill this and other similar fulltime positions.

Pyramid Consulting, Inc. provides equal employment opportunities to all employees and applicants for employment and prohibits discrimination and harassment of any type without regard to race, color, religion, age, sex, national origin, disability status, genetics, protected veteran status, sexual orientation, gender identity or expression, or any other characteristic protected by federal, state or local laws.