Databricks Solution Architect - Onsite- W2

OpenTalent; King of Prussia (PA); On-site; USD 150,000 - 190,000; Full time

Job summary

OpenTalent seeks an experienced Databricks Solution Architect to design and lead enterprise-scale data and AI solutions on the Databricks platform. You will translate business needs into scalable Lakehouse architectures using Spark, Delta Lake, and Unity Catalog.

The role requires hands-on Databricks expertise, cloud platform experience (AWS, Azure, or GCP), and governance, security, and performance optimization across development, testing, and production environments.

Qualifications

• 12+ years in Data Engineering/Architecture or related fields.
• 5+ years hands-on Databricks experience, including architecture.
• Proficient in Python and/or Scala.
• Strong SQL and performance tuning skills.
• Experience with AWS, Azure, or GCP and cloud storage.
• Terraform/IaC and CI/CD experience.
• Excellent communication and stakeholder-management skills.

Responsibilities

• Design and implement Databricks Lakehouse architectures across environments.
• Define target-state architectures for data engineering, analytics, and ML workloads.
• Architect solutions using Databricks, Delta Lake, Apache Spark, Unity Catalog, Databricks SQL, Workflows, and Lakehouse Federation.
• Design and implement enterprise data governance, security, access control, lineage, and data-sharing strategies using Unity Catalog.
• Lead migration of legacy data platforms, data warehouses, and Hive Metastore environments to Databricks/Unity Catalog.
• Develop scalable data ingestion and processing architectures for batch and streaming workloads.
• Provide technical leadership for performance tuning, scalability, reliability, and cost optimization of Databricks environments.
• Design cloud-native solutions leveraging AWS, Azure, or GCP and integrate Databricks with cloud storage, networking, security, and identity services.
• Establish IaC and deployment strategies using Terraform and CI/CD.
• Collaborate with DevOps teams to implement automated deployment pipelines and environment promotion.
• Provide architectural guidance to data engineers and development teams and conduct architecture/design reviews.
• Work with customers and stakeholders to understand requirements and present architecture options and recommendations.
• Create architecture diagrams, technical design documents, standards, and implementation roadmaps.
• Troubleshoot complex production issues and provide architectural recommendations.
• Stay current with emerging Databricks Data + AI capabilities, Generative AI, ML, and Lakehouse technologies.

Skills

Databricks

Python/Scala

SQL development

Cloud architecture

Data pipelines

Tools

Databricks Lakehouse Platform

Apache Spark

Delta Lake

Unity Catalog

Terraform

CI/CD

AWS

Azure

GCP

Power BI/Tableau

Job description

Job Title: Databricks Solution Architect

Location: King of Prussia, PA – Onsite

Experience Level: 12+Years of experience

Job Summary

We are looking for an experienced Databricks Solution Architect to design, architect, and lead the implementation of enterprise-scale data and AI solutions using the Databricks Data Intelligence Platform. The ideal candidate should have strong hands-on experience with Databricks, Lakehouse architecture, Apache Spark, Delta Lake, Unity Catalog, cloud platforms, data engineering, and modern data/AI architectures. The candidate will work closely with business stakeholders, data engineers, developers, cloud teams, and enterprise architects to translate business requirements into scalable, secure, high-performance Databricks solutions. Databricks' architecture guidance emphasizes Lakehouse architecture, governance, security, reliability, performance, cost optimization, and interoperability.

Key Responsibilities

• Design and implement enterprise-grade Databricks Lakehouse architectures across development, testing, and production environments.
• Define target-state architecture for data engineering, analytics, BI, machine learning, and AI workloads.
• Architect solutions using Databricks, Delta Lake, Apache Spark, Unity Catalog, Databricks SQL, Workflows, and Lakehouse Federation.
• Design and implement enterprise data governance, security, access control, lineage, and data-sharing strategies using Unity Catalog.
• Lead migration of legacy data platforms, data warehouses, and Hive Metastore environments to Databricks/Unity Catalog.
• Develop scalable data ingestion and processing architectures for batch and streaming workloads.
• Provide technical leadership for performance tuning, scalability, reliability, and cost optimization of Databricks environments.
• Design cloud-native solutions leveraging AWS, Azure, or GCP and integrate Databricks with cloud storage, networking, security, and identity services.
• Establish Infrastructure as Code (IaC) and deployment strategies using Terraform and CI/CD.
• Databricks currently recommends Terraform for automating workspace, networking, storage, and Unity Catalog infrastructure.
• Collaborate with DevOps teams to implement automated deployment pipelines and environment promotion.
• Provide technical guidance to data engineers and development teams and conduct architecture/design reviews.
• Work directly with customers and stakeholders to understand requirements and present architecture options and recommendations.
• Create architecture diagrams, technical design documents, standards, and implementation roadmaps.
• Troubleshoot complex production issues and provide architectural recommendations for resolution.
• Stay current with emerging Databricks Data + AI capabilities, Generative AI, ML, and Lakehouse technologies.

Required Skills

• 12+ years of experience in Data Engineering, Data Architecture, Cloud Architecture, or related fields.
• 5+ years of strong Databricks experience, including architecture and implementation.
• Strong hands-on experience with: Databricks Lakehouse Platform Apache Spark / PySpark Delta Lake Unity Catalog Databricks SQL Databricks Workflows / Jobs Delta Live Tables / Lakeflow Data governance and security
• Strong understanding of Data Lake, Data Warehouse, and Lakehouse architectures.
• Experience designing enterprise-scale batch and real-time/streaming data pipelines.
• Strong programming experience with Python and/or Scala.
• Strong SQL development and performance-tuning skills.
• Experience with at least one major cloud platform: AWS Microsoft Azure Google Cloud Platform
• Experience with cloud storage technologies such as Amazon S3, Azure Data Lake Storage, or Google Cloud Storage.
• Experience with Terraform/IaC and CI/CD.
• Strong understanding of networking, IAM, authentication, encryption, and cloud security.
• Excellent communication, presentation, documentation, and stakeholder-management skills.

Preferred Skills

• Databricks certifications are highly preferred.
• Experience with MLflow, Model Serving, Mosaic AI, and Generative AI.
• Experience with LLM/RAG architectures and AI/ML platforms.
• Experience with Kafka or other event-streaming technologies.
• Experience with data integration tools such as Informatica, ADF, dbt, or equivalent.
• Experience with Power BI/Tableau and enterprise analytics architectures.
• Experience with enterprise data governance and compliance requirements.
• Experience leading architecture workshops and technical discussions with senior stakeholders.
• Strong understanding of data modeling, dimensional modeling, metadata management, data lineage, and master data concepts.