Databricks Architect

Connect Tech+Talent; United States; On-site; USD 150,000 - 210,000; Full time

Job summary

Connect Tech+Talent seeks an experienced Databricks Architect to design, develop, and implement scalable data solutions on the Databricks Lakehouse Platform for a premier pharmaceutical sourcing and logistics enterprise. The ideal candidate will lead enterprise-level data transformation initiatives, with strong expertise in data architecture, cloud data engineering, data modeling, and governance.

You will collaborate跨跨 with data engineers, data scientists, and business stakeholders to drive

Qualifications

• Bachelor's or Master's in CS, IT, or related field.
• Strong experience with Databricks, Spark, PySpark.
• Expertise in Delta Lake and Medallion Architecture.
• Hands-on experience designing and implementing enterprise data platforms on Azure.
• Strong knowledge of data warehousing, ETL/ELT, data modeling.
• Experience with Unity Catalog, data governance, security, and access management.
• Proficiency in Python and SQL.
• Experience with CI/CD, Git, and infrastructure-as-code practices for data platforms.
• Excellent communication and technical leadership skills.

Responsibilities

• Design and implement scalable data architectures using the Databricks Lakehouse Platform.
• Develop and optimize data pipelines using PySpark, Spark SQL, and Delta Lake.
• Architect and implement Medallion Architecture for enterprise data processing.
• Integrate Databricks with Azure cloud platforms.
• Design and implement batch and streaming data processing solutions.
• Establish data governance, security, access controls, and compliance standards using Unity Catalog and related tools.
• Optimize Spark workloads, cluster configurations, query performance, and overall platform costs.
• Define data modeling, data integration, and data management strategies to support business requirements.
• Collaborate with data engineers, data scientists, business stakeholders, and enterprise architects.
• Provide technical leadership, architecture guidance, code reviews, and best practices for development teams.
• Evaluate emerging data technologies and recommend improvements to the organization's data platform.

Skills

Databricks Platform

Apache Spark

PySpark

Spark SQL

Delta Lake

Lakehouse architecture

Medallion Architecture

Azure

Data modeling

Data governance

Python

SQL

CI/CD

Infrastructure as Code

Leadership

Communication

Education

Bachelor's or Master's in CS/IT/Data Engineering

Tools

Databricks

Azure

Delta Lake

Spark SQL

Unity Catalog

Git

Terraform

Job description

We are seeking an experienced Databricks Architect for our client who is a premier global pharmaceutical sourcing

and logistics solutions enterprise ranked among the top tier of the Fortune 500. Databricks Architect will design, develop, and implement scalable, secure, and high-performance data solutions using the Databricks Lakehouse Platform. The ideal candidate will have strong expertise in data architecture, cloud data engineering, data modeling, and modern data platforms, along with experience leading enterprise-level data transformation initiatives.

Key Responsibilities

• Design and implement scalable data architectures using the Databricks Lakehouse Platform.
• Develop and optimize data pipelines using PySpark, Spark SQL, and Delta Lake.
• Architect and implement Medallion Architecture (Bronze, Silver, and Gold layers) for enterprise data processing.
• Integrate Databricks with Azure cloud platforms.
• Design and implement batch and streaming data processing solutions.
• Establish data governance, security, access controls, and compliance standards using Unity Catalog and related tools.
• Optimize Spark workloads, cluster configurations, query performance, and overall platform costs.
• Define data modeling, data integration, and data management strategies to support business requirements.
• Collaborate with data engineers, data scientists, business stakeholders, and enterprise architects.
• Provide technical leadership, architecture guidance, code reviews, and best practices for development teams.
• Evaluate emerging data technologies and recommend improvements to the organization's data platform.

Required Skills

• Bachelor's or Master's degree in Computer Science, Information Technology, Data Engineering, or a related field, or equivalent practical experience.
• Strong experience with Databricks, Apache Spark, PySpark, and Spark SQL.
• Expertise in Delta Lake, Lakehouse architecture, and Medallion Architecture.
• Hands-on experience designing and implementing enterprise data platforms on Azure.
• Strong knowledge of data warehousing, data modeling, ETL/ELT pipelines, and data integration.
• Experience with Unity Catalog, data governance, security, and access management.
• Proficiency in Python and SQL.
• Experience with performance tuning, distributed data processing, and cost optimization.
• Knowledge of CI/CD, Git, and infrastructure-as-code practices for data platforms.
• Strong communication, problem-solving, and technical leadership skills.