Lead Data Engineer - R01569363

Brillio; Bengaluru; On-site; INR 4,000,000 - 6,000,000; Full time

Job summary

Brillio in Bengaluru is seeking a Lead Data Engineer to architect and optimize Azure-based data pipelines and lakehouse architectures.

You will design scalable ETL using Data Factory, Synapse, Spark and drive real-time and batch data feeds, data modeling, and Power BI reports while ensuring data quality and security across platforms.

Qualifications

• Hands-on experience with Azure analytics and pipeline development.
• Design and implement scalable ETL processes using Azure Data Factory, Synapse Analytics, and Spark.
• Develop data pipelines for lakehouse/warehouse analytics with high performance and reliability.
• Integrate data sources from Azure Data Explorer (Kusto) and Delta Lake formats.
• Configure cloud-hosted data architectures with secure, resilient operations.
• Collaborate with cross-functional teams and drive data-driven development.
• Monitor and optimize SQL performance and data quality across systems.
• Build Power BI reports with data modeling and DAX.

Responsibilities

• Develop and maintain Azure-based data pipelines and lakehouse architectures.
• Deliver real-time and batch data feeds supporting analytics and reporting.
• Collaborate with stakeholders to capture data requirements and ensure data quality.
• Design scalable ETL processes and optimize data models for performance.
• Create dashboards and data visualizations using Power BI.

Skills

AKS

Event Hub

Azure DevOps

Azure Sentinel

Cosmos DB

Azure Functions

Spark

Pyspark

Delta Lake

SQL performance tuning

Power BI

Azure Data Lake

Azure Synapse Analytics

Azure Data Factory

Kusto

Big data

Data modeling

Education

Bachelor's degree in Computer Science, Information Technology, Data Engineering, or a closely related discipline

Microsoft Certified: Azure Data Engineer Associate

Microsoft Certified: Azure Solutions Architect Expert

Tools

Azure Synapse Analytics

Azure Data Factory

Azure Data Explorer (Kusto)

Pyspark

Delta Lake

Power BI

SQL

Job description

Lead Data Engineer

Primary Skills

• AKS, Event Hub, Azure DevOps, Azure sentinel, Cosmos DB, Azure Functions

Specialization

• Azure Data Engineering Advanced: Senior Lead Data Engineer

Job requirements

• Experience Range: with at least 4 to 6 years of hands-on experience in data engineering, including substantial expertise in Azure analytics and pipeline development
• Design and implement scalable ETL processes using Azure Data Factory, Synapse Analytics, and Spark, delivering robust real-time and batch data feeds
• Develop and optimize data pipelines and models for large-scale lakehouse and warehouse analytics systems, achieving high performance and reliability targets
• Integrate and manage data sources from Azure Data Explorer, Kusto, and Delta Lake formats to enable advanced analytics and reporting capabilities
• Configure and maintain cloud-hosted data architectures, leveraging Azure services to ensure secure, efficient, and resilient data operations
• Collaborate with cross-functional teams to capture data requirements and drive test-driven, data-driven development methodologies
• Monitor, troubleshoot, and enhance SQL performance, ensuring efficient query execution and maintaining data quality across all systems
• Develop and maintain Power BI reports, focusing on data modeling and DAX to deliver actionable insights to stakeholders
• Document technical business processes and data architecture, supporting organizational knowledge sharing and compliance
• Azure Synapse Analytics
• Azure Data Factory
• Azure Data Explorer (Kusto)
• Pyspark
• Delta Lake
• SQL performance tuning
• Power BI (Data Modeling and DAX)
• Azure Data Lake
• Azure DevOps for CI/CD
• AKS (Azure Kubernetes Service)
• Event Hub
• Azure Sentinel
• Experience with large-scale streaming analytics
• Bachelor's degree in Computer Science, Information Technology, Data Engineering, or a closely related discipline
• Microsoft Certified: Azure Data Engineer Associate
• Microsoft Certified: Azure Solutions Architect Expert
• Location: Bangalore