Lead Data Engineer - R01568940

Visa Hunt; United States; On-site; USD 150,000 - 210,000; Full time

Job summary

Brillio is seeking a Lead Data Engineer to join its Data Pipeline team. You will work on modern data architectures including lakehouse and data warehouse solutions, building real-time and batch ETL processes with Databricks, Azure Synapse, and Data Factory.

You will design scalable data models, collaborate with business, design, test and development teams, and help advance data-driven decision making at scale, leveraging AI/ML where appropriate.

Qualifications

• 5+ years’ experience with SQL required.
• 5+ years’ experience designing and implementing scalable ETL processes.
• 3+ years’ experience with modern Big Data Analytics using Data Lake, Spark and Parquet.
• 2+ years’ experience building cloud hosted data systems. Azure highly preferred.
• Building data pipelines in Azure Databricks/Fabric/Spark preferred.
• Working with data in delta lake format and from Azure Data Explorer/Kusto.
• Applying AI/ML to data engineering use cases (feature engineering, feature stores, model training/serving datasets, and model monitoring data pipelines).

Responsibilities

• Turn on Databricks, Azure Synapse Analytics and Azure Data Explorer while building ETLs and processes in Databricks and Data Factory.
• Build state-of-the-art pipelines and data models for the studio decision-making process.
• Work on large-scale lakehouse and warehouse analytics systems processing real-time and batch data.
• Collaborate with business, design, test and development teams to shape how we capture data for data-driven development.

Skills

SQL

ETL design

Data pipelines

Data modeling

Cloud architecture

Tools

Azure Databricks

Azure Synapse Analytics

Azure Data Factory

Azure Data Explorer/Kusto

Spark

Delta Lake

Job description

About Brillio:

Brillio is one of the fastest growing digital technology service providers and a partner of choice for many Fortune 1000 companies seeking to turn disruption into a competitive advantage through innovative digital adoption. Brillio, renowned for its world-class professionals, referred to as "Brillians", distinguishes itself through their capacity to seamlessly integrate cutting-edge digital and design thinking skills with an unwavering dedication to client satisfaction.

Brillio takes pride in its status as an employer of choice, consistently attracting the most exceptional and talented individuals due to its unwavering emphasis on contemporary, groundbreaking technologies, and exclusive digital projects. Brillio's relentless commitment to providing an exceptional experience to its Brillians and nurturing their full potential consistently garners them the Great Place to Work certification year after year.

Lead Data Engineer

Primary Skills

• AKS, Event Hub, Azure DevOps, Cosmos DB, Azure Functions

Specialization

• Azure Data Engineering Advanced: Senior Data Engineer

Job requirements

• Data Engineer:

  Turn 10 Studios is looking for a Data Engineer to join the Data Pipeline team. As a member of the team, you will get to live on the front edge of modern technology by building on things like Databricks, Azure Synapse Analytics and Azure Data Explorer while building ETLs and process in Databricks and Data Factory. You will help build state of the art pipelines and data models that are at the heart of the studio decision-making process. You will be working on large-scale lakehouse and warehouse analytics systems that process data feeds in real-time and batch processes. Your customers will be the business, design, test and development teams and they will look to you to help shape how we capture data to improve our test-driven methodologies and build a culture around data driven development.

  The successful candidate will bring an attention to detail along with modern engineering practices and help us build and support a high quality, modern data architecture at scale.

  Qualifications:

  5+ years’ experience with SQL required.

  5+ years’ experience designing and implementing scalable ETL processes including data movement and quality tools.

  3+ years’ experience with modern Big Data Analytics using Data Lake, Spark and formats like Parquet.

  2+ years’ experience building cloud hosted data systems. Azure highly preferred.

  Preferred:

  Building data pipelines in Azure Databricks/Fabric/Spark.

  Working with data in delta lake format and from Azure Data Explorer/Kusto

  Applying AI/ML to data engineering use cases (feature engineering, feature stores, model training/serving datasets, and model monitoring data pipelines).

  Experience preparing and governing datasets for modern AI applications (LLM/RAG, experimentation/A-B testing, and privacy-aware data access).

Know what it’s like to work and grow at Brillio: Click here

Originally posted on Himalayas