Databricks Platform Engineer – AWS, Data & DB Ops Lead

CoStar Group; Virginia (MN); On-site; USD 136,000 - 177,000; Full time

Job summary

CoStar Group is seeking a Databricks Platform & Database Engineer to own our Databricks environment on AWS and oversee a broad portfolio of relational and non-relational databases across cloud and on-premises data centers. You will drive administration, automation, governance, and platform reliability end-to-end while enabling AI-driven software delivery and observability.

Responsibilities include deploying Databricks workspaces, managing Unity Catalog, upgrades, and cost-optimized cluster

Qualifications

• Bachelor’s degree required from an accredited university or college.
• Strong track record with prior employers.
• Familiarity with LLMs, prompt engineering, or agent-based architectures.
• 5+ years in database engineering or data platform roles across relational/non-relational databases and Databricks ownership.
• Hands-on experience administering enterprise Databricks environments.
• Collaboration with cross-functional teams to ensure successful data platform implementations.
• Ability to retrieve and present critical data for ad-hoc questions.
• Knowledge of data governance and compliance best practices.
• Advanced SQL and PySpark/Python
• Terraform IaC experience; CI/CD pipelines.
• Experience with cost optimization and 24/7 operations

Responsibilities

• Deploy, administer, and maintain Databricks workspaces on AWS.
• Manage Unity Catalog objects and workspace security.
• Lead platform upgrades and environment lifecycle management.
• Monitor, optimize, and right-size clusters and SQL warehouses.
• Ingest data from relational and non-relational sources into Databricks via AWS or Lakeflow.
• Design end-to-end data workflows distributing curated data to downstream systems.
• Manage relational/non-relational databases across AWS and on-premises data centers.
• Handle provisioning, patching, backups, replication, failover, and migrations.
• Apply data security best practices including encryption and auditing.
• Develop IaC with Terraform for Databricks and RDS/Aurora resources.
• Automate cluster lifecycle, provisioning, and backup management.
• Build and maintain CI/CD for platform configuration and pipeline code.
• Implement monitoring and alerting for Databricks jobs and databases.
• Enforce cluster policies and governance for Databricks workspace.
• Manage Databricks audit logs, lineage, and compliance reporting.
• Support 24/7 production environments and lead root cause analysis.
• Collaborate with data scientists, DBAs, and DevOps to deliver platform implementations.

Skills

Databricks

AWS

PostgreSQL

MySQL

SQL Server

Python

PySpark

Terraform

CI/CD

Linux

Cost optimization

LLMs familiarity

Education

Bachelor's degree

Tools

Unity Catalog

DynamoDB

MongoDB

Looker

PowerBI

Snowflake

Terraform

Job description

CoStar Group is seeking a Databricks Platform & Database Engineer to own our Databricks environment on AWS and oversee a broad portfolio of relational and non-relational databases across cloud and on-premises data centers. You will drive administration, automation, governance, and platform reliability end-to-end while enabling AI-driven software delivery and observability.

Responsibilities include deploying Databricks workspaces, managing Unity Catalog, upgrades, and cost-optimized cluster