Senior Databricks Security Engineer & AI Automation Engineer

ECLARO; Taguig; Vor Ort; PHP 1.800.000 - 3.600.000; Vollzeit

Zusammenfassung

ECLARO is seeking a Senior Databricks Security Engineer & AI Automation Engineer to own end-to-end security for the Databricks platform, including Unity Catalog, compute, and AI workflows. You will implement network isolation, identity governance, and platform-as-code practices across Azure and AWS deployments.

You will design AI-driven security automation, integrate audit logs with SIEM, and work with finance-regulated compliance requirements to deliver measurable risk reduction for data

Qualifikationen

• 5+ years in cloud security or security engineering, with 2+ years on Databricks in production.
• Expert knowledge of Unity Catalog, access control, and encryption with CMKs.
• Experience with SSO/SCIM, OAuth, and Databricks API/SDK automation.
• Strong Python scripting and IaC experience (Terraform).
• Solid cloud networking knowledge (Azure/AWS) and private connectivity.
• Experience in regulated financial services security and governance.
• Familiarity with data protection (classification, masking, tokenization) and DLP.

Aufgaben

• Own end-to-end security architecture, configuration, and hardening of Databricks platform.
• Design network isolation, private links, and access controls across cloud deployments.
• Define identity governance, token policy, and service principals standards.
• Build Unity Catalog governance and implement row-level and column-level security.
• Enforce compute guardrails, secret scopes, and CMK-based encryption.
• Deliver platform as code with Terraform, drift detection, and auditable controls.
• Serve as security SME for new data/AI use cases and security releases.
• Develop AI automation for security workflows and posture monitoring.

Kenntnisse

Databricks security

Unity Catalog

SSO/SCIM integration

Python scripting

Terraform

Cloud networking

SIEM integration

Jira / ServiceNow

Tools

Terraform

Splunk

Databricks REST API

Jira

ServiceNow

Jobbeschreibung

Role: Senior Databricks Security Engineer & AI Automation Engineer

Responsibilities

Databricks Platform Security Architecture & Operations

• Own the end-to-end security architecture, configuration, and hardening of the Databricks platform across account and workspace administration, Unity Catalog, compute, Databricks SQL, Mosaic AI / Model Serving, and Databricks Apps.
• Design and maintain network isolation strategy, including VNet/VPC injection, secure cluster connectivity, Private Link / PrivateLink, IP access lists, serverless network policies, and egress controls across Azure and AWS deployments.
• Architect identity and access: SSO and SCIM provisioning from the corporate IdP, group-based entitlements, service principal standards, OAuth token policy, and elimination of long-lived personal access tokens.
• Build and maintain the Unity Catalog governance model (metastore, catalog, schema, and volume layout; privilege inheritance; row filters and column masks; attribute-based access control) aligned to data classification, business-unit ownership, and regulatory scope (banking regulatory, GLBA, PCI-DSS, SOX).
• Define and enforce compute guardrails through cluster policies, compute policies, init script and library allow-lists, secret scopes, and customer-managed keys for storage and workspace encryption.
• Deliver the platform as code: Terraform-managed workspaces, policies, and grants with drift detection, so every security control is reviewable, repeatable, and auditable.
• Serve as the Databricks security subject-matter expert for the security organization, reviewing new data and AI use cases, managing vendor escalations, and evaluating new Databricks features and security releases.

AI Agent Development for Data Platform Security Automation

• Design, build, and operationalize AI agents that automate routine Databricks security workflows: access request review, entitlement recertification, privilege-drift triage, secrets and token hygiene, policy exception processing, and audit evidence assembly.
• Develop prompt-engineered investigation workflows that correlate Databricks audit logs and system tables with identity data, data classification, and threat intelligence to produce prioritized, analyst-ready findings.
• Build monitoring agents that detect and triage risky behavior such as anomalous bulk data access, unauthorized external sharing, over-permissioned service principals, public or misconfigured storage locations, and policy bypass attempts.
• Create AI-driven posture agents that continuously assess workspace and account configuration against internal standards and industry benchmarks, open remediation tickets, and track them to closure.
• Build automated reporting agents that generate executive dashboards, regulatory and audit summaries, and business-unit-level data-platform risk scorecards without manual intervention.
• Secure the AI workloads themselves: define guardrails for model serving endpoints, vector search, agent frameworks, and MCP / tool integrations, including prompt injection defenses, least-privilege tool access, and output logging.
• Continuously tune agent logic based on false-positive analysis, remediation outcome data, and feedback from data engineering and data science teams to improve accuracy and reduce noise.
• Evaluate and integrate the Databricks REST API, SDK, and system tables into AI orchestration pipelines for programmatic configuration review, access analysis, and automated evidence collection.

Data Platform Security Program Operations

• Partner with data engineering, data science, analytics, and cloud platform teams to embed security into pipelines, notebooks, and model lifecycles, and to drive remediation of critical and high-risk findings within defined SLA timelines.
• Develop and maintain data-platform security KPIs, dashboards, and trend analyses (coverage of Unity Catalog, token and secret hygiene, privileged access, policy exceptions) that give leadership actionable risk visibility.
• Instrument Databricks audit logs and system tables into Splunk for correlation with broader security telemetry, enabling enriched detection, investigation, and response workflows.
• Own data protection controls on the platform in partnership with the data governance and DLP teams: classification tagging, masking and tokenization patterns, Delta Sharing and external-sharing review, and data exfiltration monitoring.
• Manage the exception and risk acceptance process for data-platform controls, ensuring documentation meets regulatory and audit requirements.
• Maintain runbooks, architecture diagrams, and operational documentation that enable the broader security team to support Databricks security operations and incident response.
• Track the data and AI security landscape (Databricks security bulletins, CVEs, CISA KEV, vendor advisories, emerging AI threats) and translate emerging risks into actionable control and monitoring priorities.

Qualifications

Required

• 5+ years in cloud security, data security, or security engineering, including 2+ years of deep, hands-on Databricks experience in a production enterprise environment.
• Expert-level knowledge of Databricks security features: Unity Catalog and fine-grained access control, workspace and account administration, cluster and compute policies, secrets management, audit logging and system tables, and encryption with customer-managed keys.
• Demonstrated experience with Databricks identity integration (SSO, SCIM, service principals, OAuth) and with the Databricks REST API or SDK for programmatic configuration, auditing, and automation.
• Strong scripting and automation skills in Python (strongly preferred) for building integrations, data pipelines, and reporting automation, plus Infrastructure as Code experience (Terraform preferred).
• Solid understanding of cloud networking and security on Azure and/or AWS: private connectivity, segmentation, storage security, key management, and IAM.
• Experience managing data security and access governance in regulated financial services environments (banking regulatory, GLBA, PCI-DSS, SOX) with demonstrated ability to drive remediation accountability.
• Working knowledge of data protection techniques (classification, masking, tokenization, DLP) and of common risks in data and analytics platforms, including over-permissioned access, credential and token sprawl, and data exfiltration.
• Experience with SIEM integration and log engineering (Splunk preferred) and with ticketing systems (ServiceNow, Jira) for workflow automation.

Preferred

• Experience building or integrating AI/LLM-driven workflows for security operations (alert triage, report generation, automated remediation orchestration).
• Familiarity with prompt engineering, AI orchestration and agent frameworks, MCP, and API-driven automation patterns for security use cases.
• Databricks certifications (Data Engineer, Machine Learning, or Platform Administrator) and relevant cloud security certifications (Azure Security Engineer, AWS Security Specialty, CCSP) or equivalent demonstrated expertise.
• Experience securing ML and GenAI workloads, including model registries, model serving endpoints, vector search, and AI gateway controls.
• Familiarity with Spark, Delta Lake, and data engineering workflows sufficient to review notebooks, jobs, and pipelines for security risk.
• Experience with Databricks Lakehouse Federation, Delta Sharing, and third-party data integrations from a security-review standpoint.