Senior Databricks Engineer

Sharp Decisions; Jersey City (NJ); Hybrid; USD 120,000 - 180,000; Full time

Benefits offered by this job

Hybrid work model

Job summary

Sharp Decisions in Jersey City, NJ, is seeking a senior data engineering contractor to design, develop, and deploy cloud data solutions within a cybersecurity data lake initiative. You will build data feeds, ensure data quality, and collaborate across IT stakeholders to align KDEs with data models.

The role emphasizes Databricks, Azure cloud services, Python/Java, and strong governance; you will mentor junior team members, participate in architecture discussions, and help scale data pipelines

Qualifications

• Azure cloud data platform experience with Databricks.
• Strong data governance and KDE/data quality knowledge.
• Experience designing data ingestion and performance tuning.

Responsibilities

• Design, develop, test, and support cloud data solutions for cybersecurity data.
• Lead data feed development and mentor junior team members.
• Coordinate integration of applications within the data ecosystem.
• Document KDEs and align to logical/physical data models.
• Collaborate with data owners to define metrics for dashboards.
• Establish data views for downstream consumption and enforce DQ rules.
• Work with scheduling and batch monitoring of production jobs.
• Resolve data quality issues and improve procedures.

Skills

Azure Databricks

Unity Catalog

Azure Data Factory

Azure Data Lake Storage

Azure Functions

Azure Event Grid

Azure Log Analytics

Development & Data Engineering

Python / Java

SQL Server

Oracle / PL-SQL

T-SQL

ETL / ELT

REST API

CI/CD

Collibra DQIM

IBM Infosphere DA

Job description

Jersey City, NJ — Hybrid (3 Days Onsite)

Contract to Hire

W2 Only — No Third Parties 1

8+ Years Experience

Azure Databricks, Python, PySpark, SQL

This role is responsible for designing, developing, and deploying cloud data solutions as part of a broader cyber data initiative focused on building out a security and risk data platform for Information Security. The engineer will develop data feeds and contribute to the buildout of a Cybersecurity Data Lake — centralizing data and establishing effective data governance around data sources and lineage using Databricks. This individual will collaborate with developers, data owners, governance leads, and business analysts across IT and aligned application stakeholders.

? Critical Requirements

• Financial services or similarly regulated environment experience strongly preferred

Responsibilities

• Design, develop, test, and support cloud data solutions focusing on data ingestion, data quality, tuning, and performance of upstream cybersecurity data sources using Databricks
• Provide indirect leadership and mentorship to junior team members; lead data feed development efforts and design initiatives
• Participate in development meetings to align priorities and objectives, assign tasks, and share experiences across applications under development
• Consult with technology and development teams to coordinate integration of applications within the broader software ecosystem
• Capture and document metadata for Key Data Elements (KDEs) to ensure accuracy and completeness for Data Quality (DQ) rules and daily dataset processing
• Work with the data architecture team to align KDEs to logical data models, develop physical data structures, and document physical data names, definitions, and types
• Partner with data owners and stakeholders to create technical requirements and DQ rules around data elements needed in the data warehouse
• Partner with Business Management and Data Owners to identify critical metrics and data fields required for metric dashboards
• Establish views to encapsulate data for downstream consumption and ensure alignment with DQ rules established on metadata
• Utilize IBM TWS production job scheduling and adhere to standards around daily scheduling and batch monitoring of production jobs
• Identify and resolve data quality issues including inaccuracies and incomplete information; implement improved procedures and processes

Qualifications & Skills

Azure & Cloud Platform

Azure Databricks & Unity Catalog

Azure Data Factory

Azure Data Lake Storage

Azure Functions

Azure Event Grid

Azure Log Analytics & Monitor

Development & Data Engineering

Python / Java

SQL Server, Oracle / PL-SQL, T-SQL

ETL / ELT

REST API & Microservices

CI/CD — Git, Jenkins, Azure DevOps

Collibra DQIM/DQE, IBM Infosphere DA

Security Domain & Compliance

• Strong knowledge of enterprise Information Security data domains: Phishing, Identity Management, Privileged Access, Cloud Security, Incident Response, Vulnerability Management, and Threat Detection
• Data knowledge of PaaS/SaaS products including ServiceNow, CrowdStrike, MS Purview, Proofpoint, WIZ.IO, JIRA, SharePoint, and Azure Active Directory
• Understanding of information security frameworks (NIST, CIS, CRI Profile) and regulatory compliance (NYSDFS, GDPR, CCPA)

? Nice to Have

• Knowledge of Microsoft Sentinel for SIEM
• Knowledge of IBM Tivoli Workload Scheduler
• Exposure to Power BI for data visualization and reporting