Senior Databricks Security & AI Automation Engineer

Rising Tide Digital, Inc.; Taguig; On-site; PHP 7,561,000 - 11,342,000; Full time

Job summary

Rising Tide Digital, Inc. seeks a Senior Databricks Platform Security Architect to own end-to-end security across Databricks accounts and workspaces.

You will design network isolation, IAM provisioning, and Unity Catalog governance while implementing policy-driven compute guardrails with drift-free Terraform code. You will lead AI agent development for automated security workflows, integrate audit data with threat intel, and collaborate with data and cloud platform teams to embed security into

Qualifications

• 5+ years in cloud security, data security, or security engineering.
• 2+ years of deep, hands-on Databricks experience in production environments.
• Expert knowledge of Unity Catalog, access control, and encryption with CMKs.
• Experience with SSO/SCIM and OAuth for programmatic config.

Responsibilities

• Own end-to-end security architecture, configuration, and hardening of the Databricks platform across accounts and workspaces.
• Design and maintain network isolation strategies, including Private Link, IP allow-lists, and egress controls.
• Architect identity and access management, including IdP provisioning and service principals.
• Build and maintain Unity Catalog governance aligned to data classification and regulatory scope.
• Define compute guardrails via policies, init scripts, secret scopes, and CMKs.
• Deliver platform-as-code with Terraform and drift detection for repeatable controls.
• Serve as security SME for Databricks features and vendor escalations.
• Design and operationalize AI agents to automate security workflows.
• Develop investigator workflows correlating audit logs with identity data and threat intel.

Skills

Cloud security

Databricks security

Identity & access management

Terraform

Python scripting

Azure/AWS networking

SIEM (Splunk)

Jira/ServiceNow integration

Tools

Databricks REST API/SDK

Splunk

Terraform

Jira

ServiceNow

Job description

Responsibilities

Databricks Platform Security Architecture & Operations

• Own the end-to-end security architecture, configuration, and hardening of the Databricks platform across account and workspace administration, Unity Catalog, compute, Databricks SQL, Mosaic AI / Model Serving, and Databricks Apps.
• Design and maintain network isolation strategy, including VNet/VPC injection, secure cluster connectivity, Private Link / PrivateLink, IP access lists, serverless network policies, and egress controls across Azure and AWS deployments.
• Architect identity and access: SSO and SCIM provisioning from the corporate IdP, group-based entitlements, service principal standards, OAuth token policy, and elimination of long-lived personal access tokens.
• Build and maintain the Unity Catalog governance model (metastore, catalog, schema, and volume layout; privilege inheritance; row filters and column masks; attribute-based access control) aligned to data classification, business-unit ownership, and regulatory scope (banking regulatory, GLBA, PCI-DSS, SOX).
• Define and enforce compute guardrails through cluster policies, compute policies, init script and library allow-lists, secret scopes, and customer-managed keys for storage and workspace encryption.
• Deliver the platform as code: Terraform-managed workspaces, policies, and grants with drift detection, so every security control is reviewable, repeatable, and auditable.
• Serve as the Databricks security subject-matter expert for the security organization, reviewing new data and AI use cases, managing vendor escalations, and evaluating new Databricks features and security releases.

AI Agent Development for Data Platform Security Automation

• Design, build, and operationalize AI agents that automate routine Databricks security workflows: access request review, entitlement recertification, privilege-drift triage, secrets and token hygiene, policy exception processing, and audit evidence assembly.
• Develop prompt-engineered investigation workflows that correlate Databricks audit logs and system tables with identity data, data classification, and threat intelligence to produce prioritized, analyst-ready findings.
• Build monitoring agents that detect and triage risky behavior such as anomalous bulk data access, unauthorized external sharing, over-permissioned service principals, public or misconfigured storage locations, and policy bypass attempts.
• Create AI-driven posture agents that continuously assess workspace and account configuration against internal standards and industry benchmarks, open remediation tickets, and track them to closure.
• Build automated reporting agents that generate executive dashboards, regulatory and audit summaries, and business-unit-level data-platform risk scorecards without manual intervention.
• Secure the AI workloads themselves: define guardrails for model serving endpoints, vector search, agent frameworks, and MCP / tool integrations, including prompt injection defenses, least-privilege tool access, and output logging.
• Continuously tune agent logic based on false-positive analysis, remediation outcome data, and feedback from data engineering and data science teams to improve accuracy and reduce noise.
• Evaluate and integrate the Databricks REST API, SDK, and system tables into AI orchestration pipelines for programmatic configuration review, access analysis, and automated evidence collection.

Data Platform Security Program Operations

• Partner with data engineering, data science, analytics, and cloud platform teams to embed security into pipelines, notebooks, and model lifecycles, and to drive remediation of critical and high-risk findings within defined SLA timelines.
• Develop and maintain data-platform security KPIs, dashboards, and trend analyses (coverage of Unity Catalog, token and secret hygiene, privileged access, policy exceptions) that give leadership actionable risk visibility.
• Instrument Databricks audit logs and system tables into Splunk for correlation with broader security telemetry, enabling enriched detection, investigation, and response workflows.
• Own data protection controls on the platform in partnership with the data governance and DLP teams: classification tagging, masking and tokenization patterns, Delta Sharing and external-sharing review, and data exfiltration monitoring.
• Manage the exception and risk acceptance process for data-platform controls, ensuring documentation meets regulatory and audit requirements.
• Maintain runbooks, architecture diagrams, and operational documentation that enable the broader security team to support Databricks security operations and incident response.
• Track the data and AI security landscape (Databricks security bulletins, CVEs, CISA KEV, vendor advisories, emerging AI threats) and translate emerging risks into actionable control and monitoring priorities.

Qualifications:

• 5+ years in cloud security, data security, or security engineering, including 2+ years of deep, hands-on Databricks experience in a production enterprise environment.
• Expert-level knowledge of Databricks security features: Unity Catalog and fine-grained access control, workspace and account administration, cluster and compute policies, secrets management, audit logging and system tables, and encryption with customer-managed keys.
• Demonstrated experience with Databricks identity integration (SSO, SCIM, service principals, OAuth) and with the Databricks REST API or SDK for programmatic configuration, auditing, and automation.
• Strong scripting and automation skills in Python (strongly preferred) for building integrations, data pipelines, and reporting automation, plus Infrastructure as Code experience (Terraform preferred).
• Solid understanding of cloud networking and security on Azure and/or AWS: private connectivity, segmentation, storage security, key management, and IAM.
• Experience managing data security and access governance in regulated financial services environments (banking regulatory, GLBA, PCI-DSS, SOX) with demonstrated ability to drive remediation accountability.
• Working knowledge of data protection techniques (classification, masking, tokenization, DLP) and of common risks in data and analytics platforms, including over-permissioned access, credential and token sprawl, and data exfiltration.
• Experience with SIEM integration and log engineering (Splunk preferred) and with ticketing systems (ServiceNow, Jira) for workflow automation.