Senior Data Platform Engineer – Databricks & AWS

ScolerTec Inc; United States; On-site; USD 140,000 - 190,000; Full time

Job summary

ScolerTec Inc. has multiple openings for Senior Data Platform Engineers – Databricks & AWS to support a large-scale federal cloud modernization and data transformation program.

The successful candidates will be responsible for designing, building, administering, optimizing, configuring, maintaining, and governing an enterprise Databricks Lakehouse Platform that supports scalable data engineering, analytics, and governance capabilities.

Qualifications

• Databricks Lakehouse Platform expertise and administration.
• Unity Catalog, cluster management, SQL warehouses, and job/workflow orchestration experience.
• AWS-based or hybrid Databricks environments familiarity; strong Python and SQL skills.

Responsibilities

• Design, configure, administer, and maintain enterprise Databricks Lakehouse environments.
• Manage Databricks workspaces, Unity Catalog, clusters, SQL warehouses, and workloads.
• Develop and optimize data pipelines using Spark, Delta Lake, DLT, and Databricks Workflows.

Skills

Databricks Lakehouse Platform

Unity Catalog

Apache Spark

Delta Lake

ETL/ELT pipelines

AWS

Python

SQL

CI/CD

Security

Platform observability

Terraform

Databricks administration

Education

MS in Computer Science or related field

Tools

Jenkins

GitLab CI

GitHub Actions

Azure DevOps

Grafana

Prometheus

Datadog

CloudWatch

Elastic

Docker

Kubernetes

Job description

ScolerTec Inc. has multiple openings for Senior Data Platform Engineers – Databricks & AWS to support a large-scale federal cloud modernization and data transformation program.

The successful candidates will be responsible for designing, building, administering, optimizing, configuring, maintaining, and governing an enterprise Databricks Lakehouse Platform that supports scalable data engineering, analytics, and governance capabilities.

This role requires strong hands-on experience with Databricks administration, Unity Catalog, Apache Spark, Delta Lake, ETL/ELT pipelines, AWS, Terraform, Python, SQL, CI/CD, performance tuning, security, and platform observability.

Key Responsibilities

• Design, configure, administer, and maintain enterprise Databricks Lakehouse environments.
• Manage Databricks workspaces, Unity Catalog, clusters, SQL warehouses, serverless capabilities, and workload management.
• Develop and optimize data pipelines and ETL/ELT processes using Apache Spark, Delta Lake, Delta Live Tables, and Databricks Workflows.
• Implement and govern Unity Catalog catalogs, schemas, tables, roles, groups, RBAC, and access controls.
• Automate Databricks provisioning, configuration, and deployments using Terraform, Python, SQL, Databricks Asset Bundles, and APIs.
• Implement platform monitoring, logging, alerting, observability, and capacity planning.
• Perform Spark and SQL performance tuning, scalability testing, and troubleshooting.
• Optimize Databricks costs through autoscaling, right-sizing, auto-termination, workload isolation, and DBU management.
• Support platform upgrades, release management, patching, and adoption of new Databricks capabilities.
• Integrate Databricks with enterprise data sources, governance tools, BI platforms, and AI/ML environments.
• Implement security, encryption, networking, auditing, compliance, high availability, and disaster-recovery controls.
• Support security audits, ATO activities, incident response, and root-cause analysis.
• Establish reusable platform standards, engineering patterns, reference architectures, runbooks, and documentation.
• Provide technical guidance and mentoring to data engineers and platform users.

Qualifications

• MA/MS with 12+ years of general experience and 10+ years of specialized experience, or equivalent experience as defined by the program.
• Strong hands-on experience implementing and administering the Databricks Lakehouse Platform.
• Deep experience with Unity Catalog, cluster management, SQL warehouses, and job/workflow orchestration.
• Strong expertise in SQL and Apache Spark performance tuning.
• Experience with Databricks performance engineering, scalability testing, and workload optimization.
• Hands‑on experience with AWS-based or hybrid Databricks environments.
• Strong Python and Bash scripting experience.
• Experience with CI/CD platforms such as Jenkins, GitLab CI, GitHub Actions, or Azure DevOps.
• Experience with Terraform or other Infrastructure as Code technologies.
• Experience with observability tools such as Grafana, Prometheus, Datadog, CloudWatch, or Elastic.
• Knowledge of ETL/ELT, distributed processing, lakehouse/data warehouse architectures, and data ingestion.
• Understanding of cloud security, IAM, access control, networking, and resource governance.
• Familiarity with Docker, Kubernetes, and microservices architectures.
• Familiarity with federal security/compliance processes such as ATO or FedRAMP is preferred.

Preferred Certifications

• Databricks Certified Data Engineer – Associate or Professional
• Databricks Certified Associate Platform Administrator
• Databricks Certified Associate Developer for Apache Spark
• AWS Certified Solutions Architect – Professional
• AWS Certified Data Analytics – Specialty
• AWS SysOps
• Docker Certified Associate
• Certified Data Management Professional (CDMP