Data Platform Engineer – Databricks & AWS

ScolerTec Inc; United States; On-site; USD 140,000 - 170,000; Full time

Job summary

ScolerTec Inc. seeks a Data Platform Engineer – Databricks & AWS to design, administer, optimize, secure, and support an enterprise Databricks Lakehouse environment. This hands-on role focuses on scalable data engineering, governance, performance, reliability, and platform operations.

The ideal candidate will have strong production experience with Databricks on AWS, Unity Catalog, clusters, SQL Warehouses, Delta Lake, Spark, Terraform, monitoring, security, and cost optimization.

Qualifications

• Must have hands-on experience deploying and administering Databricks Lakehouse platforms in production.
• Significant expertise with Unity Catalog, cluster management, SQL Warehouses, and job/workflow orchestration.
• Strong experience operating Databricks on AWS environments and performance tuning.

Responsibilities

• Design, configure, administer, and maintain enterprise Databricks Lakehouse environments.
• Manage Databricks workspaces, compute, clusters, SQL Warehouses, serverless capabilities, and workload management.
• Implement and administer Unity Catalog: catalogs, schemas, tables, groups, RBAC, permissions, and governance controls.
• Develop and optimize data pipelines using Apache Spark, Delta Lake, Delta Live Tables, and Databricks Workflows.
• Automate platform provisioning and deployment using Terraform, Python, SQL, Databricks APIs.
• Perform Spark and SQL performance tuning and scalability testing.
• Establish monitoring, logging, alerting, observability, and capacity planning.
• Optimize costs via autoscaling, right-sizing, and workload isolation.
• Integrate Databricks with AWS services, governance platforms, BI tools, and AI/ML environments.
• Implement IAM, RBAC, encryption, networking, auditing, and security controls.
• Support high availability, DR, incident response, and release management.
• Provide guidance to data engineers and platform users.

Skills

Databricks Lakehouse

Unity Catalog

SQL

Apache Spark

Python

Bash scripting

CI/CD orchestration

AWS IAM & security concepts

Data governance

Troubleshooting

Tools

Terraform

Jenkins

GitHub Actions

GitLab CI

Azure DevOps

CloudWatch

Datadog

Prometheus

Grafana

Elastic

Databricks APIs

Job description

ScolerTec is seeking Data Platform Engineer – Databricks & AWS to design, administer, optimize, secure, and support an enterprise Databricks Lakehouse environment. This role is heavily hands-on and will support scalable data engineering, analytics, governance, performance, reliability, and platform operations.

The ideal candidate has strong production experience administering Databricks on AWS, including Unity Catalog, clusters, SQL Warehouses, Workflows, Delta Lake, Spark, Terraform, monitoring, security, and performance optimization.

Key Responsibilities

• Design, configure, administer, and maintain enterprise Databricks Lakehouse environments.
• Manage Databricks workspaces, compute, clusters, SQL Warehouses, serverless capabilities, and workload management.
• Implement and administer Unity Catalog, including catalogs, schemas, tables, groups, RBAC, permissions, and governance controls.
• Develop and optimize data pipelines and ETL/ELT workloads using Apache Spark, Delta Lake, Delta Live Tables, and Databricks Workflows.
• Automate platform provisioning, configuration, and deployment using Terraform, Python, SQL, Databricks APIs, and Asset Bundles.
• Perform Spark and SQL performance tuning, scalability testing, and troubleshooting.
• Implement monitoring, logging, alerting, observability, capacity planning, and operational health dashboards.
• Optimize Databricks platform costs through autoscaling, auto-termination, right-sizing, workload isolation, and usage analysis.
• Integrate Databricks with AWS services, enterprise data sources, governance platforms, BI tools, and AI/ML environments.
• Implement IAM, RBAC, encryption, networking, auditing, security, and compliance controls.
• Support high availability, disaster recovery, incident response, root-cause analysis, upgrades, patching, and release management.
• Establish reusable platform standards, reference architectures, runbooks, and engineering practices.
• Provide technical guidance to data engineers and platform users.

Qualifications

• Strong hands-on experience implementing and administering Databricks Lakehouse Platform in production.
• Significant experience with Unity Catalog, cluster management, SQL Warehouses, and Databricks job/workflow orchestration.
• Strong experience operating Databricks in AWS environments.
• Strong SQL and Apache Spark experience, including performance tuning and workload optimization.
• Experience with Delta Lake and modern lakehouse architectures.
• Hands-on experience with Terraform or comparable Infrastructure as Code.
• Strong Python and Bash scripting experience.
• Experience with CI/CD tools such as Jenkins, GitHub Actions, GitLab CI, or Azure DevOps.
• Experience with CloudWatch, Datadog, Prometheus, Grafana, Elastic, or comparable observability tools.
• Understanding of AWS IAM, networking, access controls, security, auditing, and resource governance.
• Experience troubleshooting platform, pipeline, compute, and data-processing issues.
• Knowledge of ETL/ELT, distributed data processing, data ingestion, and lakehouse architecture.
• Federal or regulated-environment experience is preferred.

Preferred Certifications

• Databricks Certified Data Engineer Professional or Associate
• Databricks Certified Associate Platform Administrator
• Databricks Certified Associate Developer for Apache Spark
• AWS Certified Solutions Architect – Professional