Databricks Architect

CEI; Pittsburgh (Allegheny County); Hybrid; USD 140,000 - 190,000; Full time

Benefits offered by this job

Medical, Dental, Vision benefits

401(k) match

Job summary

CEI is seeking a Databricks Architect to design and implement enterprise-grade data platforms supporting analytics and AI/ML initiatives. You will balance architecture ownership with hands-on development, mentoring engineers while ensuring scalable, cost-efficient solutions across ingestion, transformation and governance.

Join CEI’s AI-first engineering team in a hybrid Pittsburgh-based role (3 days on-site, 2 remote) or fully remote within EST/CST preferences.

Qualifications

• 8–12+ years in data engineering or data architecture
• 3+ years hands-on production with Databricks
• Experience with Delta Lake, Bronze/Silver/Gold patterns
• Experience with Unity Catalog governance and data lineage
• Cloud platform experience (AWS/Azure/GCP)
• Infrastructure-as-code using Terraform
• CI/CD for data pipelines (Git-based)
• Strong data modeling and Data Vault concepts
• Cost optimization and performance tuning in Databricks

Responsibilities

• Architect end-to-end data solutions in Databricks across ingestion, transformation, governance and serving layers
• Lead hands-on development of production data pipelines with PySpark; Scala where suitable
• Define and implement Unity Catalog structures and access controls
• Collaborate with data governance to translate policies into platform controls
• Define data serving layer strategy using Databricks SQL and BI tools
• Establish platform-as-code practices with Terraform or Databricks Asset Bundles
• Design CI/CD pipelines for notebooks, jobs, and infra with Git workflows
• Define orchestration using Databricks Workflows and Lakeflow with Airflow when needed
• Oversee cost and performance governance across clusters and compute choices
• Mentor engineers; participate in architecture discussions and cross-functional forums

Skills

Databricks

PySpark

Spark SQL

Delta Lake

Unity Catalog

Terraform

CI/CD

Data modeling

ETL/ELT

cost optimization

Security & governance

Cloud platforms

Tools

Databricks Asset Bundles

Lakeflow

Job description

Back Databricks Architect
Enterprise Platforms Pittsburgh , Pennsylvania Direct Hire 100% Remote Jun 17, 2026 CEI is seeking a Databricks Architect to join our growing organization! Client/Industry: CEI – AI / IT Solutions/Consulting and Professional Services Job Title: Databricks Technical Architect Location: Hybrid in Pittsburgh, PA 15205 (3 Days On-Site / 2 Days Remote) | OR | Remote (1 st pref-EST; 2 nd pref-CST) Work Schedule/Shift : Monday to Friday, Standard business hours | Minimum expectation of ~40 work hours per week. Duration/Length of Assignment : W2 Direct Hire / Full Time Compensation : Competitive pay (Salary) | Benefits available (optional) [Medical, Dental, Vision, and 401(k) match]

Position Overview

This role supports CEI’s continued evolution into an AI-first engineering and transformation company by building and scaling enterprise-grade data platforms that enable analytics, reporting, and AI/ML initiatives. The position sits within CEI’s Solutions and Engineering organization, working alongside data engineering, analytics, governance, and business teams to deliver a modern Lakehouse platform. The individual will act as a senior technical leader, partnering across multiple stakeholders while guiding a team of data engineers and contributing directly to architectural decisions and implementation. The role operates within a collaborative, cross-functional environment where engineering standards, governance, and platform scalability are core priorities. At a high level, this individual will design and implement end-to-end data solutions, establish platform standards, and ensure the data ecosystem supports both analytical and AI-driven use cases. The expectation is to balance architecture ownership with hands‑on development, contributing directly to platform buildout, optimization, and governance. This role requires consistent collaboration with internal teams, active participation in architecture discussions, and ongoing mentorship of engineers while ensuring delivery of reliable, scalable, and cost‑efficient data solutions.

Required Skills/Experience/Qualifications

• 8–12+ years of experience in data engineering, data architecture, or related roles
• 3+ years of hands‑on production experience with Databricks
• Strong expertise in Apache Spark using PySpark and Spark SQL
• Experience designing and implementing Lakehouse architectures using Delta Lake and medallion (Bronze/Silver/Gold) patterns, with a strong grasp of partitioning, Z‑ordering, liquid clustering, and Delta optimization techniques
• Experience with Unity Catalog including governance design, access control, and data lineage
• Implementation experience on at least one major cloud platform: AWS, Azure, or GCP
• Experience implementing infrastructure‑as‑code using Terraform and/or Databricks Asset Bundles
• Experience building and managing CI/CD pipelines for data engineering workflows (Git‑based, with environment promotion)
• Strong understanding of data modeling approaches including dimensional modeling and Data Vault
• Experience with ETL/ELT patterns and distributed data systems
• Experience with cost optimization and performance tuning within Databricks environments at scale
• Understanding of data governance, security, and compliance principles and ability to implement them within platform design

Preferred Skills (Not Required)

• Scala is a plus but not required
• Databricks certifications such as Data Engineer Professional or Solution Architect
• Experience with Structured Streaming and event‑driven architectures (Kafka, Event Hubs, Kinesis, Pub/Sub)
• Experience with ingestion frameworks such as Lakeflow Connect
• Familiarity with MLflow, feature stores, and ML platform integrations
• Experience supporting BI tools such as Power BI or Tableau at scale, including semantic modeling
• Exposure to open table formats such as Iceberg or Delta UniForm and Delta Sharing.
• Experience working within regulated industries such as healthcare, financial services, or public sector

Day to Day/Responsibilities

• Architect end‑to‑end data solutions within Databricks, spanning ingestion, transformation, governance, and serving layers using the medallion (Bronze/Silver/Gold) architecture pattern on Delta Lake
• Lead hands‑on development of production data pipelines using PySpark and Spark SQL, with Scala leveraged where appropriate, ensuring code quality, scalability, and performance across workloads
• Perform code reviews and establish engineering standards through reusable frameworks and reference implementations to ensure consistency across the platform
• Own and design Unity Catalog structures, including catalog and schema organization, access controls, row‑ and column‑level security, data classification, lineage tracking, and audit capabilities
• Partner directly with data governance teams to translate business policies into enforceable platform‑level controls within Databricks
• Define and implement the data serving layer strategy using Databricks SQL, including materialized views, performance tuning, and enabling consumption through BI tools such as Power BI and Tableau, including semantic modeling
• Establish platform‑as‑code practices using Terraform and/or Databricks Asset Bundles, ensuring consistent infrastructure deployment and environment standardization
• Design and operate CI/CD pipelines for notebooks, jobs, and infrastructure using Git‑based workflows with environment promotion, supporting controlled releases across development, testing, and production environments
• Define orchestration strategies using Databricks Workflows and Lakeflow Declarative Pipelines, integrating with tools such as Airflow when cross‑platform orchestration is required
• Own cost and performance governance across the platform, including cluster policies, serverless versus classic compute decisions, Photon usage, job versus all‑purpose cluster strategy, and DBU budgeting/chargeback models
• Collaborate with AI and ML teams to support platform integration with MLflow, feature engineering on Delta Lake, and emerging Databricks AI capabilities such as vector search and Mosaic, ensuring the platform supports model development and deployment
• Provide technical leadership and mentorship to data engineers, acting as the primary Databricks subject matter expert across architecture discussions and cross‑functional forums

About Us

CEI is an AI-first engineering and transformation company that designs, builds, and operationalizes production‑grade AI and data solutions across enterprise environments. Originally established as an IT consulting and staffing firm, CEI has evolved into a technology partner focused on delivering enterprise‑scale AI transformation, modern data platforms, and application development solutions. We work across industries including healthcare, financial services, utilities, retail, and manufacturing, supporting both mid‑market and Fortune 1000 clients. CEI combines consulting, engineering, and delivery capabilities to not only define strategies but also build and implement solutions at scale. With a focus on integrating AI into core business operations, CEI enables organizations to modernize their technology landscape, improve decision‑making, and generate measurable outcomes through advanced data and AI capabilities.