Databricks Solution Architect

Astir IT Solutions, Inc.; King of Prussia (PA); On-site; USD 150,000 - 190,000; Full time

Job summary

Astir IT Solutions, Inc. seeks an experienced Databricks Solution Architect to design and lead enterprise-scale data and AI solutions using the Databricks platform. The role emphasizes Lakehouse architecture, governance, and secure, scalable implementations.

The ideal candidate brings hands-on Databricks expertise, cloud experience (AWS/Azure/GCP), and strong collaboration with stakeholders to translate requirements into architecture options and roadmaps.

Qualifications

• 12+ years of experience in Data Engineering, Data Architecture, Cloud Architecture, or related fields.
• 5+ years of strong Databricks experience, including architecture and implementation.
• Hands-on experience with Databricks Lakehouse Platform, Unity Catalog, Databricks SQL, and Databricks Workflows/Jobs.
• Strong programming experience with Python and/or Scala; strong SQL development and performance-tuning skills.
• Experience designing enterprise-scale batch and real-time data pipelines; knowledge of data governance and security.

Responsibilities

• Design and implement enterprise-grade Databricks Lakehouse architectures across development, testing, and production environments.
• Define target-state architecture for data engineering, analytics, BI, ML, and AI workloads.
• Architect solutions using Databricks, Delta Lake, Apache Spark, Unity Catalog, Databricks SQL, Workflows/Jobs, and Lakehouse Federation.
• Design and implement enterprise data governance, security, access control, lineage, and data-sharing strategies using Unity Catalog.
• Lead migration of legacy data platforms, data warehouses, and Hive Metastore to Databricks/Unity Catalog.
• Develop scalable data ingestion and processing architectures for batch and streaming workloads.
• Provide technical leadership for performance tuning, scalability, reliability, and cost optimization of Databricks environments.
• Collaborate with DevOps to implement automated deployment pipelines and IaC.

Skills

Databricks Lakehouse Platform

Unity Catalog

Databricks SQL

Databricks Workflows/Jobs

Python

Scala

SQL tuning

Data governance and security

Cloud architecture

Architectural leadership

Tools

Terraform

CI/CD

Informatica

dbt

Power BI

Tableau

Job description

Location: King of Prussia, PA – Onsite- Local profiles

Experience Level: 14+Years of experience

Job Summary

We are looking for an experienced Databricks Solution Architect to design, architect, and lead the implementation of enterprise-scale data and AI solutions using the Databricks Data Intelligence Platform. The ideal candidate should have strong hands-on experience with Databricks, Lakehouse architecture, Apache Spark, Delta Lake, Unity Catalog, cloud platforms, data engineering, and modern data/AI architectures.

The candidate will work closely with business stakeholders, data engineers, developers, cloud teams, and enterprise architects to translate business requirements into scalable, secure, high-performance Databricks solutions.

Databricks' architecture guidance emphasizes Lakehouse architecture, governance, security, reliability, performance, cost optimization, and interoperability.

Key Responsibilities

• Design and implement enterprise-grade Databricks Lakehouse architectures across development, testing, and production environments.
• Define target-state architecture for data engineering, analytics, BI, machine learning, and AI workloads.
• Architect solutions using Databricks, Delta Lake, Apache Spark, Unity Catalog, Databricks SQL, Workflows, and Lakehouse Federation.
• Design and implement enterprise data governance, security, access control, lineage, and data-sharing strategies using Unity Catalog.
• Lead migration of legacy data platforms, data warehouses, and Hive Metastore environments to Databricks/Unity Catalog.
• Develop scalable data ingestion and processing architectures for batch and streaming workloads.
• Provide technical leadership for performance tuning, scalability, reliability, and cost optimization of Databricks environments.
• Design cloud-native solutions leveraging AWS, Azure, or GCP and integrate Databricks with cloud storage, networking, security, and identity services.
• Establish Infrastructure as Code (IaC) and deployment strategies using Terraform and CI/CD. Databricks currently recommends Terraform for automating workspace, networking, storage, and Unity Catalog infrastructure.
• Collaborate with DevOps teams to implement automated deployment pipelines and environment promotion.
• Provide technical guidance to data engineers and development teams and conduct architecture/design reviews.
• Work directly with customers and stakeholders to understand requirements and present architecture options and recommendations.
• Create architecture diagrams, technical design documents, standards, and implementation roadmaps.
• Troubleshoot complex production issues and provide architectural recommendations for resolution.
• Stay current with emerging Databricks Data + AI capabilities, Generative AI, ML, and Lakehouse technologies.

Required Skills

• 12+ years of experience in Data Engineering, Data Architecture, Cloud Architecture, or related fields.
• 5+ years of strong Databricks experience, including architecture and implementation.
• Strong hands-on experience with:
• Databricks Lakehouse Platform
• Unity Catalog
• Databricks SQL
• Databricks Workflows / Jobs
• Data governance and security
• Strong understanding of Data Lake, Data Warehouse, and Lakehouse architectures.
• Experience designing enterprise-scale batch and real-time/streaming data pipelines.
• Strong programming experience with Python and/or Scala.
• Strong SQL development and performance-tuning skills.

Experience with at least one major cloud platform:

• AWS
• Experience with cloud storage technologies such as Amazon S3, Azure Data Lake Storage, or Google Cloud Storage.
• Experience with Terraform/IaC and CI/CD.
• Strong understanding of networking, IAM, authentication, encryption, and cloud security.
• Excellent communication, presentation, documentation, and stakeholder-management skills.

Preferred Skills

• Databricks certifications are highly preferred.
• Experience with MLflow, Model Serving, Mosaic AI, and Generative AI.
• Experience with LLM/RAG architectures and AI/ML platforms.
• Experience with Kafka or other event-streaming technologies.
• Experience with data integration tools such as Informatica, ADF, dbt, or equivalent.
• Experience with Power BI/Tableau and enterprise analytics architectures.
• Experience with enterprise data governance and compliance requirements.
• Experience leading architecture workshops and technical discussions with senior stakeholders.
• Strong understanding of data modeling, dimensional modeling, metadata management, data lineage, and master data concepts.

Thank you,

Bharath M

Account Manager

Email: [e-mail]

Direct : 7326946000*806