Azure Databricks Engineer

Highbrow LLC; Frisco, Omaha, Oregon, Berkeley Heights (TX, NE, WI, NJ); On-site; USD 120,000 - 150,000; Full time

Job summary

A tech solutions company is seeking an Azure Databricks Engineer to provide expertise in data analytics and Databricks development. The role involves designing data pipelines, ensuring database security, and participating in agile practices. Candidates should have over 5 years of experience with Azure Databricks and relational database technologies. This position offers long-term employment and requires local presence with no relocation.

Qualifications

• 5+ years experience with Azure Databricks and relational database technologies.
• 5+ years experience designing data pipelines in Azure Databricks.
• Understanding of Medallion Architecture using Azure Services.

Responsibilities

• Provide expertise on Azure Databricks development tasks.
• Assist with Azure Databricks deployments, testing, and configuration.
• Create detailed documentation for the database.

Skills

Azure Databricks Analytics

Azure Data Lake Storage Gen 2

Azure Event Hubs

Python

Terraform

Azure Networking Services

Tools

Azure Data Studio

Visual Studio

Azure CLI

Job description

Job Title: Azure Databricks Engineer

Job ID: 2024-13177

Job Location: Berkeley Heights, NJ, Omaha, NE, or Frisco, TX (100% Onsite – local only no relocation)

Job Travel Location(s):

# Positions: 1

Employment Type: W2

Duration:Long Term

# of Layers:0

Work Eligibility:US Permanent Eligibility to Work Required

Key Technology:Azure Databricks

Job Responsibilities:

• Understand all concepts covered in Azure Data Engineer Associate, Azure Database Administrator Associate, Microsoft Azure Data Fundamentals and Power Bl Data Analyst Associate certifications.
• Provide expertise and ownership of Azure Databricks development tasks within the scrum team. Interacts effectively with clients and leadership, and can adapt communication for the appropriate audience.
• Reads and comprehends software requirements, assisting with development of agile user stores and tasks. Assists with troubleshooting configuration and performance issues.
• Assists with Azure Databricks deployments, testing, configuration, and installation.
• Ensures security is a priority and understands the various areas where security vulnerabilities arise with database technologies.
• Ensures database resiliency, and disaster recovery capabilities.
• Comprehensive understanding of backups, restores, redundancy, geo-replication and Azure storage concepts. Documentation and Knowledge Sharing, creates detailed documentation for the database. Facilitates knowledge sharing within the team for effective collaboration.

Skills and Experience Required:

Required:

• 5+ years proven experience working with Azure Databricks Analytics database capabilities, specifically Azure Databricks and other relational database technologies supported in Azure.
• 5+ years proven experience with Azure Data Lake Storage Gen 2, Azure Databricks, Azure Data Explorer, Azure Event Hubs, Spark Pools, Python, PySpark, SOL, Azure Landing Zone, Azure Networking Services, Microsoft EntralD.
• 5+ years proven experience with Azure geo-redundancy, HA/failover technologies.
• 5+ years proven experience designing and implementing data pipelines using Azure Databricks for data cleaning, transformation, and loading into Data Lakehouse.
• 5+ years proven experience with Infrastructure as Code {laC) tools such as Terraform.
• 5+ years proven experience with programming languages such as Python, PySpark and data constructs such as JSON or XML.
• Understanding of implementing Medallion Architecture using Azure Services. Strong experience with relational and/or dimensional data modeling.

Desired:

• Experience in large scale cloud data migrations using Databricks, Python, Spark, SOL. Understanding of Azure DevOps tools in an Azure production setting.
• Strong experience with Azure Data Studio, Visual Studio database management/projects/design capabilities, Azure CLI, and Powershell.
• Strong and current familiarity with Azure platforms and services, including Azure App Services, Azure API Management, Azure Monitor, Azure Application Insights, Azure SOL, etc.
• Experience with Cl/CD pipelines and deployment automation using tools like Azure DevOps. Experience with Agile methodologies.
• Strong understanding of security best practices for Azure database deployments, including, access controls, 1AM, data encryption (TOE), Azure key vaults and masking.
• Awareness of FedRAMP, FISMA, and/or Zero Trust Architecture strongly desired. Experience with SOL Server versions 2005 – 2022 is a plus.
• Experience with Oracle or Microsoft Sql Server, either in Azure or on-prem, is a plus.
• Preferred Certifications:Azure Data Engineer Associate, Azure Database Administrator Associate, Azure Data Fundamentals, Power Bl Data Analyst Associate