Databricks Lakehouse Engineer – AWS Platform Expert

ScolerTec Inc; United States; On-site; USD 140,000 - 170,000; Full time

Job summary

ScolerTec Inc. seeks a Data Platform Engineer – Databricks & AWS to design, administer, optimize, secure, and support an enterprise Databricks Lakehouse environment. This hands-on role focuses on scalable data engineering, governance, performance, reliability, and platform operations.

The ideal candidate will have strong production experience with Databricks on AWS, Unity Catalog, clusters, SQL Warehouses, Delta Lake, Spark, Terraform, monitoring, security, and cost optimization.

Qualifications

• Must have hands-on experience deploying and administering Databricks Lakehouse platforms in production.
• Significant expertise with Unity Catalog, cluster management, SQL Warehouses, and job/workflow orchestration.
• Strong experience operating Databricks on AWS environments and performance tuning.

Responsibilities

• Design, configure, administer, and maintain enterprise Databricks Lakehouse environments.
• Manage Databricks workspaces, compute, clusters, SQL Warehouses, serverless capabilities, and workload management.
• Implement and administer Unity Catalog: catalogs, schemas, tables, groups, RBAC, permissions, and governance controls.
• Develop and optimize data pipelines using Apache Spark, Delta Lake, Delta Live Tables, and Databricks Workflows.
• Automate platform provisioning and deployment using Terraform, Python, SQL, Databricks APIs.
• Perform Spark and SQL performance tuning and scalability testing.
• Establish monitoring, logging, alerting, observability, and capacity planning.
• Optimize costs via autoscaling, right-sizing, and workload isolation.
• Integrate Databricks with AWS services, governance platforms, BI tools, and AI/ML environments.
• Implement IAM, RBAC, encryption, networking, auditing, and security controls.
• Support high availability, DR, incident response, and release management.
• Provide guidance to data engineers and platform users.

Skills

Databricks Lakehouse

Unity Catalog

SQL

Apache Spark

Python

Bash scripting

CI/CD orchestration

AWS IAM & security concepts

Data governance

Troubleshooting

Tools

Terraform

Jenkins

GitHub Actions

GitLab CI

Azure DevOps

CloudWatch

Datadog

Prometheus

Grafana

Elastic

Databricks APIs

Job description

ScolerTec Inc. seeks a Data Platform Engineer – Databricks & AWS to design, administer, optimize, secure, and support an enterprise Databricks Lakehouse environment. This hands-on role focuses on scalable data engineering, governance, performance, reliability, and platform operations.

The ideal candidate will have strong production experience with Databricks on AWS, Unity Catalog, clusters, SQL Warehouses, Delta Lake, Spark, Terraform, monitoring, security, and cost optimization.