Databricks Engineer

IDR, Inc.; United States; Remote; CAD 168,000 - 251,000; Full time

Job summary

IDR, Inc. is seeking a Data Engineer (Databricks) to join a top client in a remote, direct-hire role.

Focused on developing and optimizing an enterprise lakehouse on Azure Databricks, the candidate will own multiple Databricks domains and leverage Databricks Genie for conversational analytics. The ideal candidate has 4+ years of Databricks experience, including data ingestion, security, governance, and data quality, and can independently own Databricks solutions while guiding best practices

Qualifications

• 4+ years hands-on Azure Databricks experience.
• Experience across multiple Databricks areas beyond pipelines.
• Hands-on with Databricks Genie for conversational analytics.
• Experience preparing data for AI initiatives.

Responsibilities

• Develop, maintain, and optimize an enterprise lakehouse on Azure Databricks.
• Own ingestion, pipelines, transformations, security, governance, and data quality.
• Leverage Databricks Genie to enable natural-language data exploration and AI-driven analytics.
• Collaborate with a small team to establish Databricks best practices.

Skills

Python

T-SQL

ANSI SQL

Agile methodologies

DevOps concepts

Git

AI-driven analytics

Tools

Azure Databricks

Databricks Genie

Job description

IDR is seeking a Data Engineer (Databricks) to join one of our top clients in a remote, direct-hire opportunity. This role is focused on the continued development and optimization of an enterprise lakehouse built on Azure Databricks. The ideal candidate has deep, hands‑on Databricks expertise, experience across multiple areas of the platform, and a strong understanding of leveraging Databricks Genie and AI-driven analytics to support business initiatives.

• Location: Remote – USA (MT, CT, or ET) or Ontario, Canada

Position Overview for the Data Engineer (Databricks):

• Develop, maintain, and optimize an enterprise lakehouse architecture using Azure Databricks.
• Own multiple aspects of the Databricks environment, including data ingestion, pipelines, transformations, security, governance, and data quality.
• Leverage Databricks Genie (conversational analytics) to improve data accessibility and support AI-driven business initiatives.
• Prepare, structure, and optimize enterprise data for AI and natural-language analytics applications.
• Identify opportunities to leverage AI for process automation, workflow optimization, and innovative data solutions beyond code generation.
• Collaborate with a small team of data engineers and architects to develop scalable solutions and establish Databricks best practices.

Requirements for the Data Engineer (Databricks):

• 4+ years of hands‑onAzure Databricks experience, with deep knowledge of its tools, features, and capabilities.
• Demonstrated experience across multiple areas of Databricks beyond pipeline development, including ingestion, security, user access, data quality, and governance.
• Experience developing, maintaining, and optimizing enterprise lakehouse environments.
• Hands‑on experience with Databricks Genie for conversational analytics, including enabling natural-language queries and AI-driven data exploration.
• Experience preparing and transforming data to support AI initiatives, with an understanding of how data should be structured for AI consumption.
• Proven ability to independently own Databricks solutions and explain specific technical implementations, processes, and tools used.

Nice to Have:

• 5+ years of overall data engineering or data warehousing experience.
• Strong Python, T-SQL, and ANSI SQL skills.
• Experience with Agile methodologies, DevOps concepts, and Git.
• Experience leveraging AI for business process automation and workflow improvements.
• Background working on smaller data engineering teams with broad technical responsibilities.