Azure Databricks Data Engineer

TALENT Software Services; Louisville (KY); On-site; USD 110,000 - 150,000; Full time

Job summary

Talent Software Services is seeking a Databricks / Azure Data Engineer in Louisville, KY for a 6-month engagement. You will design and maintain data pipelines using Azure Data Factory, ingest data from multiple sources, and perform transformations with Azure Databricks and Delta Lake.

You will implement and maintain Prefect workflows, use Azure DevOps for CI/CD, participate in Agile ceremonies, and collaborate with development teams to ensure data quality and reliable pipelines.

Qualifications

• Experience developing and implementing data pipelines using Azure Data Factory (ADF).
• Experience ingesting data from multiple sources using Azure ADF.
• Experience transforming data using Azure Databricks and Delta Lake.
• Experience with Azure DevOps for CI/CD, including stories, bugs and issue management.
• Experience implementing Prefect.
• Strong hands-on experience with Databricks.
• Experience working with Microsoft Azure integration solutions.

Responsibilities

• Work on Agile development projects and participate in sprint delivery.
• Design and implement data integration solutions in Microsoft Azure.
• Develop and maintain data pipelines using Azure Data Factory.
• Ingest data from various source systems into Azure.
• Perform data transformations using Azure Databricks and Delta Lake.
• Implement and maintain workflows using Prefect.
• Perform data analysis and troubleshooting for data-related issues.
• Assist in identifying and resolving data quality and pipeline issues.
• Collaborate with development teams to support CI/CD processes using Azure DevOps.
• Act as an individual contributor and problem solver.
• Participate in Agile ceremonies, sprint planning, and delivery activities.

Skills

Azure Data Factory

Azure Databricks

Delta Lake

Azure DevOps

Prefect

Databricks

Tools

Azure Data Factory

Azure Databricks

Delta Lake

Prefect

Job description

Job Responsibilities

• Experience developing and implementing data pipelines using Azure Data Factory (ADF).
• Experience ingesting data from multiple sources using Azure ADF.
• Experience transforming data using Azure Databricks and Delta Lake.
• Experience with Azure DevOps for CI/CD, including:
  • Stories
  • Bugs
  • Issue management
• Experience implementing Prefect.
• Strong hands‑on experience with Databricks.
• Experience working with Microsoft Azure integration solutions.
• Work on Agile development projects and participate in sprint delivery.
• Design and implement data integration solutions in Microsoft Azure.
• Ingest data from various source systems into Azure.
• Perform data transformations using Azure Databricks and Delta Lake.
• Implement and maintain workflows using Prefect.
• Perform data analysis and troubleshooting for data-related issues.
• Assist in identifying and resolving data quality and pipeline issues.
• Collaborate with development teams to support CI/CD processes using Azure DevOps.
• Act as an individual contributor and problem solver.
• Participate in Agile ceremonies, sprint planning, and delivery activities.

Job Requirements

• Location: Louisville, KY
• Duration: 6 months
• Experience Required: 8-10 years
• Role: Databricks / Azure Data Engineer

Must-Have Technical & Functional Skills

• Experience developing and implementing data pipelines using Azure Data Factory (ADF).
• Experience ingesting data from multiple sources using Azure ADF.
• Experience transforming data using Azure Databricks and Delta Lake.
• Experience with Azure DevOps for CI/CD, including:
  • Stories
  • Bugs
  • Issue management
• Experience implementing Prefect.
• Strong hands‑on experience with Databricks.
• Experience working with Microsoft Azure integration solutions.

Roles & Responsibilities

• Work on Agile development projects and participate in sprint delivery.
• Design and implement data integration solutions in Microsoft Azure.
• Develop and maintain data pipelines using Azure Data Factory.
• Ingest data from various source systems into Azure.
• Perform data transformations using Azure Databricks and Delta Lake.
• Implement and maintain workflows using Prefect.
• Perform data analysis and troubleshooting for data-related issues.
• Assist in identifying and resolving data quality and pipeline issues.
• Collaborate with development teams to support CI/CD processes using Azure DevOps.
• Act as an individual contributor and problem solver.
• Participate in Agile ceremonies, sprint planning, and delivery activities.

Generic Managerial / Agile Skills

• Experience working in Agile development environments.
• Experience with sprint planning and sprint delivery.
• Strong analytical and problem‑solving skills.
• Ability to work independently as an individual contributor.

Role Details

• Role Description: Databricks
• Essential Skill: Databricks
• Desirable Skills: Not specified
• Keyword: Databricks
• Skill Category: Digital - Databricks