Senior Data Engineer – Azure Databricks / PySpark

TALENT Software Services; Minnetonka Mills (MN); On-site; USD 110,000 - 140,000; Full time

Job summary

TALENT Software Services is seeking a Data Engineer / API Role with AI Builder in Minnesota to design, build, and optimize cloud-based data pipelines on Azure. You will work with Databricks, ADF, PySpark, and Delta Lake to deliver scalable solutions and integrated APIs.

The role emphasizes end-to-end delivery, data governance, and security, with an on-site setup in Minnesota for an experienced professional.

Qualifications

• Experience designing scalable data pipelines on Azure.
• Experience building REST/GraphQL APIs and deploying on Kubernetes/AKS.
• Exposure to AI/LLM technologies and Databricks AI/ML?
• Knowledge of Unity Catalog and data governance.

Responsibilities

• Own end-to-end solution delivery for data and API workloads.
• Analyze and optimize data pipelines for performance and reliability.
• Collaborate with cross-functional teams and stakeholders.
• Build scalable data pipelines and cloud-based solutions on Azure.
• Implement enterprise governance, security, and deployment standards.
• Optimize API performance and governance for enterprise use.

Skills

Databricks

Azure Data Factory

PySpark

Python

SQL

GitHub

CI/CD

Azure Cloud

Delta Lake

DDD/Data Modeling

Tools

Azure Databricks

PySpark

Scala

Azure Data Factory (ADF)

ADLS Gen2

Delta Lake

Unity Catalog

Python

SQL

GitHub

CI/CD

Kubernetes

AKS

Job description

• Primary Role: Data Engineer / API Role with AI Builder

Role Summary

• Experience Required: 4-6 years
• Primary Role: Data Engineer / API Role with AI Builder
• Cloud Platform: Azure Cloud
• Role Type: Individual Contributor

Core Technical Skills

• Azure Databricks
• PySpark
• Scala
• Azure Data Factory (ADF)
• Azure Data Lake Storage Gen2 (ADLS Gen2)
• Delta Lake
• Unity Catalog
• Python
• SQL
• GitHub
• CI/CD

Data Engineering Responsibilities

• Design, develop, and optimize scalable data pipelines.
• Build cloud-based data solutions using Azure.
• Develop ETL/ELT data processing workflows.
• Work with large-scale datasets in enterprise environments.
• Perform data modeling and performance tuning.
• Implement and maintain Delta Lake solutions.
• Work with Databricks, ADF, and PySpark for data engineering workloads.
• Develop end-to-end data solutions independently.
• Apply data governance and security best practices.

API Development Skills

• Strong hands-on experience building REST or GraphQL APIs.
• Understanding of:
  • API design and contracts
  • API security
  • Error handling
  • API performance optimization
• Experience deploying APIs using Kubernetes, preferably Azure Kubernetes Service (AKS).

AI / ML Skills

• Exposure to AI/LLM technologies.
• Experience or exposure to agentic frameworks.
• Familiarity with:
  • Databricks AI/ML
  • Codex
  • Claude
  • Other LLM/AI development frameworks

Cloud & DevOps

• Strong hands-on experience with Azure Cloud.
• Experience with CI/CD practices.
• GitHub experience.
• Knowledge of cloud-native data platforms.
• Understanding of Kubernetes/AKS deployment.

Data Governance & Security

• Strong understanding of data governance.
• Knowledge of data security and access control.
• Experience with Unity Catalog is preferred.
• Ability to implement secure and governed data solutions.

Essential Skills

• Databricks
• Azure Data Factory (ADF)
• PySpark
• Azure Cloud
• ADLS Gen2
• Delta Lake

Desirable Skills

• Scala
• REST APIs / GraphQL
• Kubernetes / AKS
• AI/LLM
• Agentic frameworks
• Databricks AI/ML
• CI/CD
• Unity Catalog

Key Responsibilities

• Own end-to-end solution delivery.
• Analyze and solve complex data engineering problems.
• Collaborate with cross-functional teams and business stakeholders.
• Build scalable and reliable data pipelines.
• Optimize data processing and API performance.
• Follow enterprise standards for governance, security, and deployment.