AWS Databricks Platform Administrator

Extend Information Systems Inc.; Raritan (NJ); Vor Ort; USD 120.000 - 190.000; Vollzeit

Zusammenfassung

Extend Information Systems Inc. in Raritan, NJ seeks an experienced AWS Databricks Platform Administrator to own platform reliability, security, and cost management across analytics workloads.

You will manage Databricks on AWS, Unity Catalog, and associated services, ensuring scalable, secure operations. The role requires strong Databricks administration skills, AWS knowledge, and production-support experience, with automation using Terraform and Python.

Qualifikationen

• 9 years of overall experience in cloud infrastructure, data platforms, DevOps, or production administration.
• 5 years of hands-on Databricks administration, preferably enterprise deployments on AWS.
• Strong knowledge of Databricks account console, workspace administration, Unity Catalog, clusters, SQL warehouses, jobs, and audit logs.
• Strong knowledge of AWS networking, security, storage, encryption, monitoring, and identity services.
• Hands-on IaC and automation using Terraform, Python, REST APIs, and CICD tools.
• Experience supporting production platforms with incident response, capacity planning, and SLA governance.
• Bachelor's degree or equivalent experience; Databricks account and Unity Catalog administration.

Aufgaben

• Administer and support Databricks accounts, workspaces, Unity Catalog, metastores, clusters, and jobs on AWS.
• Design secure, scalable, highly available Databricks platform architecture aligned with cloud standards.
• Manage identity and access via IAM roles, SCIM, and RBAC.
• Implement data governance using Unity Catalog including catalogs, schemas, and auditability.
• Configure AWS integrations such as S3, VPC, KMS, CloudWatch, CloudTrail, and Secrets Manager.
• Establish policies, tagging, and workload isolation; monitor health, performance, and capacity.
• Drive FinOps through DBU analysis, cost controls, and optimization of compute resources.
• Automate provisioning and policy enforcement using Terraform, Databricks CLI/SDK, and CICD pipelines.
• Lead incident, problem, change, release, vulnerability, and service requests with SLAs.
• Plan runtime upgrades, migrations, DR tests, backups, and continuity activities.
• Collaborate with data engineering, security, networking, and cloud teams to onboard workloads.
• Maintain runbooks, SOPs, and knowledge articles; provide technical leadership and roadmaps.

Kenntnisse

AWS S3

Databricks

Redshift

IAM

Terraform

Python

REST APIs

Ausbildung

Bachelor's degree in Computer Science, Information Technology, Engineering, or related field

Tools

Databricks CLI/SDK

Terraform

Python

Jobbeschreibung

Role: AWS Databricks Platform Administrator

Location: Raritan, NJ

Job Summary

We are seeking an experienced AWS Databricks Platform Administrator with 9 years of overall IT experience including substantial handson expertise in administering securing optimizing and supporting Databricks on AWS The role will own platform reliability governance security cost management automation and operational excellence across enterprise data and analytics workloads The ideal candidate combines strong Databricks administration skills with deep knowledge of AWS services infrastructure automation cloud security and production support

Skills

Mandatory Skills : AWS S3, Databricks, Redshift

Key Responsibilities

• Administer and support Databricks accounts workspaces Unity Catalog metastores clusters SQL warehouses jobs instance pools repositories libraries secrets and access controls on AWS
• Design and maintain secure scalable and highly available Databricks platform architecture aligned with enterprise cloud standards
• Manage identity and access through IAM roles and policies service principals single signon SCIM provisioning and rolebased access control
• Implement data governance using Unity Catalog including catalogs schemas external locations storage credentials privileges lineage and auditability
• Configure and troubleshoot AWS integrations such as S3 IAM VPC PrivateLink KMS CloudWatch CloudTrail Route 53 and Secrets Manager
• Establish cluster policies approved runtime standards library controls tagging autotermination serverless usage guidelines and workload isolation
• Monitor platform health performance job reliability capacity and usage proactively identify and resolve bottlenecks and recurring issues
• Drive FinOps practices by analyzing DBU and AWS consumption enforcing tagging and budgets optimizing compute configurations and reducing idle or inefficient resources
• Automate workspace provisioning configuration policy enforcement and operational tasks using Terraform Databricks Asset Bundles Databricks CLISDK Python and CICD pipelines
• Lead incident problem change release vulnerability and servicerequest management for the platform while meeting agreed SLAs and operational KPIs
• Plan and execute runtime upgrades platform migrations disasterrecovery tests backup validation and businesscontinuity activities
• Partner with data engineering analytics security networking cloud engineering and application teams to onboard workloads and resolve complex technical issues
• Maintain architecture diagrams runbooks standard operating procedures support documentation compliance evidence and knowledge articles
• Provide technical leadership mentor administrators and engineers conduct reviews and define platform standards and product roadmaps
• Coordinate with Databricks and AWS support teams for critical incidents product defects and service improvements

Required Experience and Qualifications

• 9 years of overall experience in cloud infrastructure data platforms DevOps platform engineering or production administration
• 5 years of handson Databricks administration experience preferably with enterprise deployments on AWS
• Strong experience with Databricks account console workspace administration Unity Catalog cluster policies SQL warehouses jobs Delta Lake system tables and audit logs
• Strong knowledge of AWS networking security storage encryption monitoring and identity services particularly VPC S3 IAM KMS PrivateLink CloudWatch and CloudTrail
• Handson experience with infrastructure as code and automation using Terraform Python shell scripting REST APIs Databricks CLISDK and CICD tools
• Proven experience supporting businesscritical platforms in production including incident response rootcause analysis change management capacity planning and SLA governance
• Working knowledge of Linux administration networking DNS firewalls certificates proxies and secure connectivity patterns
• Good understanding of Spark architecture Delta Lake workload performance data lakehouse patterns and troubleshooting distributed data workloads
• Experience implementing security controls leastprivilege access secrets management encryption audit logging vulnerability remediation and compliance requirements
• Bachelors degree in Computer Science Information Technology Engineering or a related discipline or equivalent relevant experience
• Databricks Account and workspace administration Unity Catalog Delta Lake SQL warehouses clusters jobs notebooks repos instance pools secrets system tables audit logs and cost controls
• AWS S3 IAM VPC EC2 KMS PrivateLink CloudWatch CloudTrail Route 53 Secrets Manager Lambda and AWS Organizations
• Operations Monitoring observability performance tuning capacity management FinOps compliance disaster recovery