Databricks Data Platform Architect

UsefulBI Corporation; Raleigh (NC); On-site; USD 130,000 - 180,000; Full time

Job summary

A leading technology firm is seeking a Databricks Data Platform Architect to design and manage their enterprise lakehouse platform on AWS. This role emphasizes DevOps and Infrastructure Engineering to ensure the reliability and scalability of the Databricks environment. Responsibilities include implementing CI/CD processes, optimizing performance, and establishing data engineering standards. Ideal candidates will have extensive experience with Databricks and strong skills in Apache Spark, Python, SQL, and AWS services.

Qualifications

• 8+ years in Data Engineering/Platform Engineering/Cloud Infrastructure.
• 4+ years hands-on with Databricks in production.
• Strong expertise in Apache Spark, Python, and SQL.
• Deep knowledge of AWS services including S3 and IAM.

Responsibilities

• Architect and manage Databricks on AWS end-to-end.
• Implement CI/CD for notebooks, jobs, and pipelines.
• Design lakehouse architecture (bronze/silver/gold).
• Drive cost optimization and compute efficiency strategies.

Skills

Apache Spark

Python

SQL

AWS

Terraform

CI/CD

Performance tuning

Tools

Databricks

CloudWatch

Datadog

Job description

Databricks Data Platform Architect (DevOps & Infrastructure Focus)

Location: Alameda, CA – Onsite (5 days/week)

About the Role

We are looking for a hands‑on Databricks Data Platform Architect to design, build, and operate our enterprise lakehouse platform on AWS.

This role combines DevOps, Infrastructure Engineering, and Data Engineering architecture.

You will own the reliability, scalability, security, and performance of our Databricks environment and enable data teams to build high-quality, production‑grade pipelines.

If you enjoy building platforms from the ground up, automating everything, and solving performance problems at scale, this role is for you.

What You’ll Own

Platform & Infrastructure Architecture (Core Focus)

• Architect and manage Databricks on AWS end-to-end
• Implement CI/CD for notebooks, jobs, and pipelines
• Define cluster policies, autoscaling, and environment isolation (dev/stage/prod)
• Set up monitoring, logging, and alerting (CloudWatch/Datadog)
• Drive cost optimization and compute efficiency strategies
• Ensure high availability, reliability, and production uptime
• Design lakehouse architecture (bronze/silver/gold)
• Optimize Spark/Delta workloads for performance and scalability
• Establish data engineering standards and best practices
• Support batch and streaming workloads
• Troubleshoot performance bottlenecks and production incidents

Security, Governance & Compliance

• Implement role-based access control using Unity Catalog
• Secure S3, IAM, and external locations
• Enforce data access policies and PII protection
• Enable audit logging and compliance guardrails
• Partner with security and governance teams to ensure best practices

Required Qualifications

• 8+ years in Data Engineering / Platform Engineering / Cloud Infrastructure
• 4+ years hands‑on with Databricks in production
• Strong expertise in Apache Spark, Python, and SQL
• Deep knowledge of AWS (S3, IAM, VPC, CloudWatch, networking)
• Experience with Terraform or other IaC tools
• CI/CD implementation experience for data platforms
• Strong troubleshooting and performance tuning skills

Preferred

• Experience with Delta Lake optimization (ZORDER, OPTIMIZE, partitioning)
• Workflow orchestration tools (Airflow/Databricks Workflows)
• Experience with enterprise data governance tools (Unity Catalog/Atlan/Collibra)
• Knowledge of GDPR/HIPAA or regulated environments
• Prior experience building platforms for multiple teams

What Success Looks Like

• Clear environment separation and automation
• Self‑service data platform for engineers and analysts
• Strong security and governance with minimal friction