Senior AI/ML Data Engineer – Databricks

Sharp Decisions; New York (NY); Hybrid; USD 180,000 - 240,000; Full time

Benefits offered by this job

Hybrid work model

Job summary

Sharp Decisions in NYC is seeking a VP, Senior Data Engineer to lead cloud data solutions for the CyberDW data lakehouse within the Information Security Data Operations group.

You will drive AI-powered analytics using Databricks, MLFlow and cloud services, ensuring governance and scalable data platforms for cybersecurity use cases while mentoring junior staff and collaborating with data owners and governance leads.

Qualifications

• Requires a bachelor’s degree in a related field and 10+ years in IT development or data governance.
• Proven experience with data platforms, governance, and secure data architecture in regulated environments.
• Strong knowledge of CI/CD and DevOps tooling and SQL-based data tooling.
• Exposure to AI/ML lifecycle, Databricks and cloud-native services is preferred.

Responsibilities

• Design, develop, test and support cloud data solutions for the CyberDW catalogue focusing on ingestion, quality, tuning and performance of security data sources.
• Lead AI lifecycle efforts including feature engineering, model development and monitoring using Databricks and MLFlow.
• Apply generative AI and LLMs to enhance detection, classification, prioritization and response across security workflows.
• Ensure governance, explainability and data controls for enterprise-scale cybersecurity data environments.
• Mentor junior team members and coordinate data-feed development with stakeholders across IT and ISDAD.

Skills

Data governance

Data modeling

CI/CD

DevOps tooling

SQL/PL-SQL

OpenAI tools exposure

Enterprise security data

PaaS/SaaS data sources

REST APIs

ETL/ELT

AI lifecycle (Databricks)

PowerBI

Data lakehouse (CyberDW)

Governance & data lineage

Problem solving

Education

Bachelor's degree in Computer Science, Information Security, Data Management or related field

Tools

Collibra DQIM/DQE

IBM Infosphere DA

Git

Jenkins

Azure DevOps

Databricks

MLFlow

Job description

Job level:Director
Location: NYC
hybrid on-site
perm FTE

Role Description

The VP, Senior Data Engineer, is a critical role within the Information Security department and will report to the Director, Information Security Data Operations

This VP role will be responsible for designing, developing & deploying cloud data solutions for ISDAD. This

is part of the overall cyber data initiative focusing on building out the security data platform for Information Security.

This individual will lead the design and implementation of AI-driven solutions on Databricks, applying machine learning, generative AI, and advanced analytics to drive automation, intelligence, and operational efficiency across multiple cybersecurity use cases. This individual will also be responsible for developing the data feeds and will be a part of the larger development effort of building out a Cybersecurity Data Lakehouse (CyberDW).

The goal of the CyberDW is to centralize the ISDAD data as well as establish effective data governance around the data sources and its data lineage. This role will collaborate with the developers, data owners, governance leads and business analysts within the Information Technology (IT) department as well as other stakeholders aligned with the applications owned by ISDAD.

Role Objectives

• Design, develop, testing & support of cloud data solutions for the CyberDW catalogue focusing on data ingestion, data quality, data tuning and performance of upstream cybersecurity data sources
• Apply hands-on expertise across the AI lifecycle, including feature engineering, model development, evaluation, deployment, and monitoring using Databricks, MLFlow, and cloud-native services.
• Leverage generative AI, LLMs, and AI-assisted decisioning to enhance detection, classification, prioritization, and response across data protection, insider risk, and security operations workflows.
• Ensure AI solutions are designed with appropriate governance, explainability, model risk awareness, and data controls to support regulated, enterprise-scale cybersecurity environments.
• Indirect leadership and mentorship of junior team members. Able to lead data feed development efforts and design initiatives. Participate in development meetings to align development priorities and objectives, assign tasks, and share experiences and challenges with applications under development.

Qualifications and Skills

• Bachelor's degree in Computer Science, Information Security, Data Management, or related field
• 10+ years' experience in IT development, data governance, data architecture or related roles, preferably in a highly regulated environment such as financial services.
• Proficient in data management & data modeling tools (e.g., Collibra DQIM/DQE, IBM Infosphere DA).
• Strong knowledge of CI/CD and DevOps tooling (i.e., Git, Jenkins, Azure DevOps)
• Proficient in SQL Server, Oracle / PL-SQL, T-SQL and SQL stored procedures
• Exposure to Open AI tools (i.e., Anthropic Claude models) is a plus
• Strong knowledge of enterprise Information Security data (i.e., Phishing, Identity Management, Privileged Access, Cloud Security, Incident Response, Vulnerability Management, Threat Detection).
• Data knowledge of PaaS/SaaS products (i.e., ServiceNow, Crowdstrike, MS Purview, Proofpoint, WIZ.IO, JIRA, SharePoint, Azure Active Directory, SAI360). Knowledge of Microsoft Sentinel for security information and event management (SIEM) is a plus.
• Understanding of information security frameworks and security controls (i.e., NIST, CIS, CRI Profile) and regulatory compliance (i.e., NYSDFS, GDPR, CCPA).
• Experience with REST API web services and microservice architecture. Strong understanding of ETL/ELT. Knowledge of IBM Tivoli Workload Schedulera plus.
• Exposure to PowerBI for data visualization and reporting is a plus.
• Problem solving and analytical skills, with an initiative-taking and results oriented approach.