Databricks Platform Architect (AWS)

CCS Global Tech; Reston (VA); On-site; USD 150,000 - 210,000; Full time

Job summary

CCS Global Tech in Reston, VA, seeks a Senior Cloud Architect to design end-to-end Databricks on AWS platform environments (dev/test/prod) with strong focus on Unity Catalog, data governance, and vaulting across network isolation, IAM, and FinOps.

You will lead implementation from scratch, using Terraform and the Databricks provider, build platform-as-code, coordinate with security and network teams, and document architecture, DR/HA, observability, and handover runbooks.

Qualifications

• 8+ years in cloud architecture with Databricks on AWS.
• Hands-on experience standing up Databricks accounts and workspaces from scratch.
• Deep knowledge of AWS VPC, PrivateLink, IAM, S3, and KMS.
• Unity Catalog design and administration.
• Identity provisioning: SSO/MFA, SCIM, group-based access, automated deprovisioning.
• Service principal lifecycle: provisioning, least privilege, authentication, revocation.
• Terraform, including the Databricks provider.
• Databricks compute architecture: classic and serverless.
• Ability to brief engineers and executives in client reviews.

Responsibilities

• Design the account and workspace topology mapped to environments (dev/test/prod) and data domains.
• Design Unity Catalog foundation: metastore strategy, catalog structure per environment, storage credentials, external locations, audit tables.
• Design network and workspace isolation: VPC, subnets, PrivateLink, transit/peering; prevent unwanted BU routing and public exposure.
• Own compute strategy across classic and serverless: optimization, scalability, reliability, cost controls, FinOps tagging.
• Implement identity: SSO/MFA, SCIM provisioning, group-based access, deprovisioning.
• Establish service principal and workload identity: automated provisioning, least privilege, authentication, auditing, revocation.
• Design encryption and key management, including customer-managed keys.
• Build the platform as code with the Databricks Terraform provider and Asset Bundles for CI/CD.
• Run proof-of-control tests and quantify results: provisioning and revocation time, isolation, network exposure.
• Work with client network/identity/security teams and support architecture reviews through to approval.
• Document reference architecture, DR/HA, observability, and handover runbook.

Skills

Cloud architecture

Databricks on AWS

Executive briefing

Tools

Databricks Terraform provider

Terraform

Databricks Asset Bundles

AWS IAM S3 KMS

Job description

What you will do

1. Design the account and workspace topology, mapped to environments (dev/test/prod), business units, and data domains, with the isolation and governance model between them.
2. Design the Unity Catalog foundation: metastore strategy, catalog structure per environment, storage credentials, external locations, and audit through system tables.
3. Design network and workspace isolation: VPC, subnet strategy, PrivateLink, and transit/peering, preventing unintended BU-to-BU routing and public exposure.
4. Own compute strategy across classic and serverless: optimization, scalability, reliability, cost controls, and FinOps tagging and reporting.
5. Implement identity: SSO/MFA, SCIM provisioning, group-based access, and automated deprovisioning.
6. Establish service principal and workload identity: automated provisioning, least privilege, authentication, auditing, and revocation.
7. Design encryption and key management, including customer-managed keys.
8. Build the platform as code with the Databricks Terraform provider, and Databricks Asset Bundles for CI/CD, so environments are repeatable and auditable.
9. Run proof-of-control tests and quantify results: provisioning and revocation time, isolation, and network exposure.
10. Work with the client’s network, identity, and security teams, and support their architecture and security reviews through to approval.
11. Document the reference architecture, DR/HA approach, observability model, and handover runbook.

What you need

• 8+ years in cloud architecture, 3+ on Databricks on AWS.
• Hands-on experience standing up Databricks accounts and workspaces from scratch in enterprise environments, not only working within an existing setup.
• AWS at depth: VPC, PrivateLink, IAM, S3, and KMS.
• Unity Catalog design and administration.
• Identity provisioning: SSO/MFA, SCIM, group-based access, automated deprovisioning.
• Service principal lifecycle: provisioning, least privilege, authentication, revocation.
• Terraform, including the Databricks provider.
• Databricks compute architecture: classic and serverless, optimization, cost controls.
• Experience designing and running proof-of-control tests with quantified outcomes.
• Ability to brief engineers and executives, and to hold design positions in client review forums.

Nice to have

• Databricks Cloud Platform Architect accreditation or Platform Administrator certification.
• AWS Solutions Architect Professional or Security Specialty.