Azure Databricks Engineer — Cybersecurity Data Lake

Sharp Decisions; Jersey City (NJ); Hybrid; USD 140,000 - 180,000; Full time

Job summary

Sharp Decisions seeks an experienced cloud data engineer to design, develop, and deploy data solutions for a cybersecurity data lake project. You will centralize data sources, establish data governance, and collaborate with data owners, governance leads, and analysts to build scalable feeds and KDE documentation.

The role emphasizes ingestion, data quality, and performance tuning within a Databricks-based environment, working closely with IT stakeholders and application teams.

Qualifications

• Azure Databricks and Unity Catalog required.
• Python and Java programming experience required.
• ETL/ELT development experience.
• REST API and microservices experience.
• Collibra DQIM/DQE and IBM Infosphere DA knowledge.
• Security domain knowledge and regulatory compliance experience.
• Knowledge of enterprise information security data domains (Phishing, Identity Management, Privileged Access, Cloud Security, Incident Response, Vulnerability Management, Threat Detection).
• Experience with PaaS/SaaS products (ServiceNow, CrowdStrike, MS Purview, Proofpoint, WIZ.IO, JIRA, SharePoint, Azure Active Directory).
• Understanding of security frameworks (NIST, CIS, CRI Profile) and regulatory compliance (NYSDFS, GDPR, CCPA).

Responsibilities

• Design, develop, test, and support cloud data solutions focusing on ingestion, quality, tuning, and performance of cybersecurity data sources using Databricks.
• Provide mentorship to junior team members and lead data feed development efforts and design initiatives.
• Participate in development meetings to align priorities, assign tasks, and share experiences across applications under development.
• Coordinate integration of applications within the broader software ecosystem with technology teams.
• Capture and document metadata for KDEs to ensure data quality rules and daily dataset processing.
• Work with data architecture to align KDEs to data models and document physical data names and types.
• Collaborate with data owners to create technical requirements and DQ rules for data warehouse elements.
• Identify metrics and data fields for dashboards with business stakeholders.
• Create views to encapsulate data for downstream consumption ensuring alignment with DQ rules.
• Utilize IBM TWS production scheduling and daily batch monitoring standards.
• Identify and resolve data quality issues and implement improved procedures.

Skills

Azure Databricks

Unity Catalog

Python

Java

ETL/ELT

REST API

Microservices

Collibra DQIM/DQE

IBM Infosphere DA

Security Domain

ServiceNow

CrowdStrike

MS Purview

Proofpoint

WIZ.IO

JIRA

SharePoint

Azure Active Directory

Tools

Databricks

Unity Catalog

IBM Infosphere DA

Collibra DQIM/DQE

Power BI

Job description

Sharp Decisions seeks an experienced cloud data engineer to design, develop, and deploy data solutions for a cybersecurity data lake project. You will centralize data sources, establish data governance, and collaborate with data owners, governance leads, and analysts to build scalable feeds and KDE documentation.

The role emphasizes ingestion, data quality, and performance tuning within a Databricks-based environment, working closely with IT stakeholders and application teams.