Azure Databricks Engineer – GenAI

BlackCube Labs; Minneapolis (MN); On-site; USD 120,000 - 180,000; Full time

Job summary

BlackCube Labs in Hartford, CT / Minnetonka, MN is seeking an AI Builder / Senior Software Engineer specializing in Databricks and GenAI. You will design, build, and maintain scalable data pipelines and AI-powered enterprise applications using .NET, Azure, and Generative AI technologies.

Lead modernization of data platforms, ensuring governance, security, and reliable ingestion/transformation. Collaborate with cross-functional teams to enable AI use cases, analytics, and actionable insights for

Qualifications

• Design, build, and maintain scalable, secure, and high performance batch and streaming data pipelines using modern data engineering practices.
• Lead the modernization and migration of on premise data platforms to cloud native architectures, owning infrastructure, networking, security, and deployment pipelines.
• Develop and maintain core data platform components that support reliable ingestion, transformation, enrichment, and consumption of enterprise data assets.
• Collaborate with data domain experts across Operations, Product, Finance, Value Analytics, and other domains to understand data sources and design efficient, connected data models.
• Ensure data accuracy, quality, relevance, and governance by implementing validation, monitoring, documentation, and security controls.
• Analyze large datasets to identify trends, patterns, and actionable insights; perform data analysis to support analytics, AI use cases, and business strategy.
• Enable reporting and analytics by making curated, trusted datasets available to BI and downstream consumers.
• Partner with AI/ML Engineers and Leads to design, build, and improve machine learning models from scratch or enhance existing models.
• Contribute to experimentation with novel ML and Generative AI architectures.
• Help design and implement AI powered enterprise applications
• Analyzes and investigates

Responsibilities

• Design, develop, and maintain scalable enterprise applications and AI-powered solutions using .NET, Azure, and Generative AI technologies.
• Build and integrate AI capabilities such as conversational assistants, intelligent search, summarization, content generation, and workflow automation into business applications.
• Implement secure, reliable, and high-performing software solutions while adhering to enterprise architecture, compliance, and governance standards.
• Drive continuous improvement through automation, AI-assisted development practices, code quality enhancements, and operational excellence initiatives

Skills

Azure Databricks

GenAI technologies

LLMs

RAG

AI Agents

Vector Databases

Prompt Engineering

Tools

.NET

Python

PySpark

Job description

Job Title: Azure Databricks Engineer – GenAI

Location: Hartford, CT 06103 or Minnetonka, MN 55345 (Onsite)

Duration: 06 Months

Must Have Skills

• Azure Databricks, with good knowledge of Generative AI technologies, including LLMs, RAG, AI Agents, Vector Databases, and Prompt Engineering

Nice to Have Skills

• .Net

Detailed Job Description

AI Builder / Senior Software Engineer (Databricks & GenAI)

Required Skills & Attributes

• Design, build, and maintain scalable, secure, and high performance batch and streaming data pipelines using modern data engineering practices
• Lead the modernization and migration of on premise data platforms to cloud native architectures, owning infrastructure, networking, security, and deployment pipelines
• Develop and maintain core data platform components that support reliable ingestion, transformation, enrichment, and consumption of enterprise data assets
• Collaborate with data domain experts across Operations, Product, Finance, Value Analytics, and other domains to understand data sources and design efficient, connected data models
• Ensure data accuracy, quality, relevance, and governance by implementing validation, monitoring, documentation, and security controls
• Analyze large datasets to identify trends, patterns, and actionable insights; perform data analysis to support analytics, AI use cases, and business strategy
• Enable reporting and analytics by making curated, trusted datasets available to BI and downstream consumers
• Partner with AI/ML Engineers and Leads to design, build, and improve machine learning models from scratch or enhance existing models
• Contribute to experimentation with novel ML and Generative AI architectures
• Help design and implement AI powered enterprise applications
• Analyzes and investigates

Preferred Skills & Attributes

• 1 year of experience working with Large Language Models (LLMs), Retrieval-Augmented Generation (RAG), AI Agents, Vector Databases, and Prompt Engineering.
• Experience developing AI-driven solutions using GitHub Copilot, Microsoft 365 Copilot, Azure AI Studio, and enterprise GenAI platforms.
• Experience with healthcare domain

Primary Responsibilities

• Design, develop, and maintain scalable enterprise applications and AI-powered solutions using .NET, Azure, and Generative AI technologies.
• Build and integrate AI capabilities such as conversational assistants, intelligent search, summarization, content generation, and workflow automation into business applications.
• Implement secure, reliable, and high-performing software solutions while adhering to enterprise architecture, compliance, and governance standards.
• Drive continuous improvement through automation, AI-assisted development practices, code quality enhancements, and operational excellence initiatives

Minimum Years of Experience:

• 4 to 6 years

Top 3 responsibilities:

• Azure Databricks
• PySpark
• Python