Databricks Engineer

HireFlex; Bengaluru; On-site; INR 3,000,000 - 6,000,000; Full time

Job summary

HireFlex is seeking a Data Engineer with strong Azure data engineering, Databricks and data lake experience to support IT Security use cases. The role involves creating scalable pipelines, ingestion, transformation, and collaboration with Data Scientists on ML-ready datasets.

You will work across security data, analytics and GenAI use cases, applying PySpark, Python and SQL to build impactful data solutions and dashboards.

Qualifications

• 8+ years of experience in Data Engineering or related field.
• Hands-on with Microsoft Azure data services and ADLS Gen2.
• Proficient with Azure Databricks and PySpark.
• Strong SQL and Python skills and understanding of security data use cases.
• Experience with Power BI dashboards and reporting.

Responsibilities

• Design, develop and maintain scalable data pipelines for IT Security and Cybersecurity use cases.
• Build and manage data solutions using ADLS Gen2, Databricks and Azure data services.
• Ingest, transform and process security data from multiple sources (SIEM, endpoint, identity, network, cloud).
• Develop Databricks/PySpark solutions for processing security telemetry at scale.
• Collaborate with Data Scientists to prepare datasets for ML and analytics.
• Support GenAI/LLM use cases involving security data and analytics.

Skills

Data engineering

Azure

Python

SQL

Databricks

GenAI

Security analytics

Tools

Azure Databricks

PySpark

Power BI

Job description

Role Overview

We are looking for a Data Engineer with strong experience in Azure data engineering, Databricks and data lake technologies, along with an understanding of IT Security/Cybersecurity use cases. The ideal candidate is a generalist who can work across security data engineering, analytics, Data Science and emerging GenAI use cases.

Key Responsibilities

• Design, develop and maintain scalable data pipelines for IT Security and Cybersecurity use cases.
• Build and manage data solutions using Azure Data Lake Storage (ADLS/ADLS Gen2), Azure Databricks and Azure data services.
• Ingest, transform and process security data from multiple sources such as SIEM, endpoint, identity, network, cloud and application security platforms.
• Develop Databricks/PySpark solutions for processing and analysing large volumes of security telemetry.
• Work with security logs, alerts, vulnerability data, threat intelligence, authentication data and other security datasets.
• Work with Azure Security services and understand security-related data generated across the Azure/Microsoft security ecosystem.
• Develop data models and curated datasets to support Security Operations, Threat Detection, Risk Analytics and reporting.
• Collaborate with Data Scientists to prepare datasets for machine learning, anomaly detection, behavioural analytics and predictive use cases.
• Support GenAI/LLM/RAG use cases involving security data, knowledge retrieval and intelligent security analytics.
• Develop automation scripts using Python Build and maintain Power BI dashboards and reports for security metrics, trends, risks and operational KPIs.
• Implement data quality, governance, access controls and monitoring across data pipelines.
• Troubleshoot data ingestion, transformation and pipeline issues and optimise performance and cost.
• Collaborate with Cybersecurity, SOC, Data Engineering, Data Science and Cloud Engineering teams.

Required Skills & Experience

• 8+ years of experience in Data Engineering, Data Analytics, Security Data Engineering or a related field.
• Strong hands-on experience with Microsoft Azure data services. Strong experience with Azure Data Lake Storage (ADLS/ADLS Gen2).
• Hands-on experience with Azure Databricks and PySpark.
• Strong SQL and Python skills. Understanding of IT Security/Cybersecurity data or security-related use cases.
• Good understanding of Azure Security / Microsoft security ecosystem. Experience in data ingestion, transformation, data modelling and pipeline development.
• Experience with Power BI dashboards and reporting.

Data Science / AI Exposure

• Good understanding of Data Science concepts, including statistics, machine learning, anomaly detection and feature engineering.
• Ability to work with Data Scientists and understand ML/data requirements.
• Exposure to GenAI, LLM, RAG or AI-powered analytics is preferred. Ability to explore and analyze large security datasets and identify meaningful patterns and trends.

Preferred Certifications:

• Databricks Certified Data Engineer Associate / Professional or equivalent Databricks certification.
• Microsoft security certifications such as SC-200 Security Operations Analyst Associate, SC-500 - Cloud and AI Security Engineer Associate
• Other relevant certifications such as Security+, CEH, CISSP or equivalent are an advantage.