Remote Azure Databricks Architect for AI Data Platform

Clearstead; United States; Remote; USD 180,000 - 240,000; Full time

Job summary

Clearstead Advisors is seeking an experienced Databricks Solution Architect to lead the design, development, and ongoing management of our enterprise data and AI platform. As a founding member of our AI & Data team, this individual will establish architecture standards, develop scalable data solutions, and ensure the security, reliability, and performance of our Databricks environment.

This highly technical, hands-on role combines leadership with data engineering responsibilities, collaborating

Qualifications

• Bachelor's degree or equivalent in a related field; advanced degree preferred.
• CDMP or Databricks certification preferred.
• 10+ years in data engineering/architecture; 4+ years in production Databricks.
• Expert in Azure Databricks, Unity Catalog, Delta Lake and data architecture.
• Hands-on PySpark, SQL, data pipelines, and integration experience.
• Experience supporting AI/ML workloads in Databricks; Azure security knowledge.
• Strong communication to translate tech for business stakeholders.

Responsibilities

• Lead architecture and development of Azure Databricks platform (Delta Lake, Unity Catalog, medallion).
• Establish standards, security controls, and scalable data integration frameworks.
• Oversee platform performance, monitoring, readiness, and cost optimization.
• Manage transition of platform ownership from external partners.
• Design, build, and maintain production data pipelines with PySpark, SQL, Databricks Workflows.
• Integrate data from Salesforce, Orion, Addepar, PARis, and other systems.
• Develop data transformation, reconciliation, and QA processes.
• Implement automated testing, deployment, and infrastructure management.
• Support AI applications, intelligent agents, and advanced analytics.
• Collaborate with AI Engineer and Data Management Lead.

Skills

Azure Databricks

PySpark

SQL

Data pipelines

Data architecture

Leadership

Communication

Education

Bachelor's degree in Data, Information Systems, Computer Science, Business Analytics, or related field

Advanced degree in Information Systems, Data Analytics, or related discipline

CDMP or Databricks certification preferred

Tools

Unity Catalog

Delta Lake

Azure Databricks

Terraform

CI/CD practices

Job description

Clearstead Advisors is seeking an experienced Databricks Solution Architect to lead the design, development, and ongoing management of our enterprise data and AI platform. As a founding member of our AI & Data team, this individual will establish architecture standards, develop scalable data solutions, and ensure the security, reliability, and performance of our Databricks environment.

This highly technical, hands-on role combines leadership with data engineering responsibilities, collaborating