Lead Data Engineer: Azure Data & Real-Time Pipelines

Visa Hunt; United States; On-site; USD 150,000 - 210,000; Full time

Job summary

Brillio is seeking a Lead Data Engineer to join its Data Pipeline team. You will work on modern data architectures including lakehouse and data warehouse solutions, building real-time and batch ETL processes with Databricks, Azure Synapse, and Data Factory.

You will design scalable data models, collaborate with business, design, test and development teams, and help advance data-driven decision making at scale, leveraging AI/ML where appropriate.

Qualifications

• 5+ years’ experience with SQL required.
• 5+ years’ experience designing and implementing scalable ETL processes.
• 3+ years’ experience with modern Big Data Analytics using Data Lake, Spark and Parquet.
• 2+ years’ experience building cloud hosted data systems. Azure highly preferred.
• Building data pipelines in Azure Databricks/Fabric/Spark preferred.
• Working with data in delta lake format and from Azure Data Explorer/Kusto.
• Applying AI/ML to data engineering use cases (feature engineering, feature stores, model training/serving datasets, and model monitoring data pipelines).

Responsibilities

• Turn on Databricks, Azure Synapse Analytics and Azure Data Explorer while building ETLs and processes in Databricks and Data Factory.
• Build state-of-the-art pipelines and data models for the studio decision-making process.
• Work on large-scale lakehouse and warehouse analytics systems processing real-time and batch data.
• Collaborate with business, design, test and development teams to shape how we capture data for data-driven development.

Skills

SQL

ETL design

Data pipelines

Data modeling

Cloud architecture

Tools

Azure Databricks

Azure Synapse Analytics

Azure Data Factory

Azure Data Explorer/Kusto

Spark

Delta Lake

Job description

Brillio is seeking a Lead Data Engineer to join its Data Pipeline team. You will work on modern data architectures including lakehouse and data warehouse solutions, building real-time and batch ETL processes with Databricks, Azure Synapse, and Data Factory.

You will design scalable data models, collaborate with business, design, test and development teams, and help advance data-driven decision making at scale, leveraging AI/ML where appropriate.