Staff AI & Data Engineer (Databricks) — Remote

Eliassen Group; Montpelier (VT); Remote; USD 110,000 - 124,000; Part time

Benefits offered by this job

Medical, Dental, Vision

401k with matching

Life insurance

Job summary

Eliassen Group is seeking a Staff AI & Data Engineer to design and build enterprise-grade data and AI solutions on Databricks. This hands-on contract role focuses on strengthening the Databricks platform, onboarding data sources, and creating reusable ingestion and transformation frameworks.

The ideal candidate has advanced Databricks, Spark, Delta Lake, and SQL expertise, plus Python, AWS data services, and experience building scalable data pipelines.

Qualifications

• Advanced hands-on experience with Databricks, Apache Spark, Delta Lake, and SQL.
• Hands-on experience with AWS services related to data analytics and AI, including EMR and Bedrock.
• Strong Python and SQL development skills.
• Experience building scalable batch and/or streaming data pipelines.
• Deep understanding of Bronze/Silver/Gold medallion architectures.
• Experience with Databricks Lakeflow or comparable orchestration technologies.
• Experience implementing enterprise data quality, lineage, observability, and monitoring.
• Experience with Unity Catalog and governed enterprise data environments.
• Strong understanding of dimensional modeling, data products, semantic models, and cross-domain integration.
• Experience with Git-based development, CI/CD, automated testing, and production deployment.
• Ability to troubleshoot complex distributed data and pipeline issues.
• Experience building AI and GenAI solutions against enterprise data.
• Understanding of semantic layers and natural-language-to-data architectures.
• Experience with Databricks Genie or comparable natural-language analytics technology.
• Experience building tools, functions, APIs, or services consumed by AI agents.
• Understanding of MCP, agent tool calling, tracing, evaluation, and AI governance.
• Experience developing production AI solutions with security, observability, and governance controls.
• Preferred: Databricks Data Engineer Professional or equivalent experience.
• Preferred: Experience with Databricks AI/BI, Genie, MLflow, and MCP-based integrations.
• Preferred: Experience in Real Estate, Finance, People/Workplace, Operations, or similar domains.
• Preferred: Experience building metadata-driven or configuration-driven ingestion frameworks.

Responsibilities

• Design and build production-grade data pipelines on Databricks.
• Ingest data from enterprise apps, databases, APIs, SaaS, files, and sources.
• Build Bronze, Silver, and Gold medallion architectures.
• Implement incremental processing and change data capture patterns.
• Standardize types, codes, units, keys, and definitions across source systems.
• Implement deduplication, historical tracking, schema evolution, and data lineage.
• Optimize pipelines for reliability, performance, cost.
• Create a configuration-driven source onboarding framework and Lakeflow templates.
• Automate source registration, deployment, monitoring, lineage, and controls.
• Implement data quality, anomaly detection, monitoring, quarantine.
• Implement job and pipeline monitoring, alerting, logging, dashboards.
• Support automated deployment and promotion using Databricks Asset Bundles and CI/CD.
• Develop conformed dimensions and shared-key strategies across domains.
• Build integrated Gold-layer products and reusable data contracts.
• Develop AI-ready Gold data products with ownership, lineage, definitions, and quality information.
• Develop semantic context, business glossary, ontology, and certified KPIs in Unity Catalog.
• Build governed Unity Catalog functions and agent-ready tools and endpoints.
• Implement tracing, evaluation, testing, monitoring, and governance for AI and agent workflows.
• Support development of AI agents that use trusted enterprise data.

Skills

Databricks

Apache Spark

Delta Lake

SQL

Python

AWS EMR

CI/CD

Git-based development

Data quality

Unity Catalog

Databricks Lakeflow

GenAI

MCP

Tools

Databricks Lakeflow

Databricks Genie

MLflow

Job description

Eliassen Group is seeking a Staff AI & Data Engineer to design and build enterprise-grade data and AI solutions on Databricks. This hands-on contract role focuses on strengthening the Databricks platform, onboarding data sources, and creating reusable ingestion and transformation frameworks.

The ideal candidate has advanced Databricks, Spark, Delta Lake, and SQL expertise, plus Python, AWS data services, and experience building scalable data pipelines.