Director, AI-Driven Cyber Data Platform Databricks

Sharp Decisions; New York (NY); Hybrid; USD 180,000 - 240,000; Full time

Benefits offered by this job

Hybrid work model

Job summary

Sharp Decisions in NYC is seeking a VP, Senior Data Engineer to lead cloud data solutions for the CyberDW data lakehouse within the Information Security Data Operations group.

You will drive AI-powered analytics using Databricks, MLFlow and cloud services, ensuring governance and scalable data platforms for cybersecurity use cases while mentoring junior staff and collaborating with data owners and governance leads.

Qualifications

• Requires a bachelor’s degree in a related field and 10+ years in IT development or data governance.
• Proven experience with data platforms, governance, and secure data architecture in regulated environments.
• Strong knowledge of CI/CD and DevOps tooling and SQL-based data tooling.
• Exposure to AI/ML lifecycle, Databricks and cloud-native services is preferred.

Responsibilities

• Design, develop, test and support cloud data solutions for the CyberDW catalogue focusing on ingestion, quality, tuning and performance of security data sources.
• Lead AI lifecycle efforts including feature engineering, model development and monitoring using Databricks and MLFlow.
• Apply generative AI and LLMs to enhance detection, classification, prioritization and response across security workflows.
• Ensure governance, explainability and data controls for enterprise-scale cybersecurity data environments.
• Mentor junior team members and coordinate data-feed development with stakeholders across IT and ISDAD.

Skills

Data governance

Data modeling

CI/CD

DevOps tooling

SQL/PL-SQL

OpenAI tools exposure

Enterprise security data

PaaS/SaaS data sources

REST APIs

ETL/ELT

AI lifecycle (Databricks)

PowerBI

Data lakehouse (CyberDW)

Governance & data lineage

Problem solving

Education

Bachelor's degree in Computer Science, Information Security, Data Management or related field

Tools

Collibra DQIM/DQE

IBM Infosphere DA

Git

Jenkins

Azure DevOps

Databricks

MLFlow

Job description

Sharp Decisions in NYC is seeking a VP, Senior Data Engineer to lead cloud data solutions for the CyberDW data lakehouse within the Information Security Data Operations group.

You will drive AI-powered analytics using Databricks, MLFlow and cloud services, ensuring governance and scalable data platforms for cybersecurity use cases while mentoring junior staff and collaborating with data owners and governance leads.