Databricks Architect: Remote Lakehouse & AI Platform Lead

CEI; Pittsburgh (Allegheny County); Hybrid; USD 140,000 - 190,000; Full time

Benefits offered by this job

Medical, Dental, Vision benefits

401(k) match

Job summary

CEI is seeking a Databricks Architect to design and implement enterprise-grade data platforms supporting analytics and AI/ML initiatives. You will balance architecture ownership with hands-on development, mentoring engineers while ensuring scalable, cost-efficient solutions across ingestion, transformation and governance.

Join CEI’s AI-first engineering team in a hybrid Pittsburgh-based role (3 days on-site, 2 remote) or fully remote within EST/CST preferences.

Qualifications

• 8–12+ years in data engineering or data architecture
• 3+ years hands-on production with Databricks
• Experience with Delta Lake, Bronze/Silver/Gold patterns
• Experience with Unity Catalog governance and data lineage
• Cloud platform experience (AWS/Azure/GCP)
• Infrastructure-as-code using Terraform
• CI/CD for data pipelines (Git-based)
• Strong data modeling and Data Vault concepts
• Cost optimization and performance tuning in Databricks

Responsibilities

• Architect end-to-end data solutions in Databricks across ingestion, transformation, governance and serving layers
• Lead hands-on development of production data pipelines with PySpark; Scala where suitable
• Define and implement Unity Catalog structures and access controls
• Collaborate with data governance to translate policies into platform controls
• Define data serving layer strategy using Databricks SQL and BI tools
• Establish platform-as-code practices with Terraform or Databricks Asset Bundles
• Design CI/CD pipelines for notebooks, jobs, and infra with Git workflows
• Define orchestration using Databricks Workflows and Lakeflow with Airflow when needed
• Oversee cost and performance governance across clusters and compute choices
• Mentor engineers; participate in architecture discussions and cross-functional forums

Skills

Databricks

PySpark

Spark SQL

Delta Lake

Unity Catalog

Terraform

CI/CD

Data modeling

ETL/ELT

cost optimization

Security & governance

Cloud platforms

Tools

Databricks Asset Bundles

Lakeflow

Job description

CEI is seeking a Databricks Architect to design and implement enterprise-grade data platforms supporting analytics and AI/ML initiatives. You will balance architecture ownership with hands-on development, mentoring engineers while ensuring scalable, cost-efficient solutions across ingestion, transformation and governance.

Join CEI’s AI-first engineering team in a hybrid Pittsburgh-based role (3 days on-site, 2 remote) or fully remote within EST/CST preferences.