Senior AI & Data Engineer - Databricks (Remote, W2)

Eliassen Group; Hartford (CT); Remote; USD 110,000 - 124,000; Full time

Benefits offered by this job

Medical benefits

401k match

Life insurance

Dental benefits

Vision benefits

Job summary

Eliassen Group is seeking a Staff AI & Data Engineer to design and build enterprise-grade data and AI solutions on Databricks. This hands-on role focuses on strengthening the Databricks platform, onboarding new data sources, and creating scalable ingestion and transformation frameworks.

The position is remote for a U.S.-based client under a W2 contract. You will work across data pipelines, semantic layers, and AI agent-ready capabilities, delivering Gold-layer data products and governance.

Qualifications

• Advanced hands-on experience with Databricks, Apache Spark, Delta Lake, and SQL.
• Hands-on experience with AWS services related to data analytics and AI, including EMR and Bedrock.
• Strong Python and SQL development skills.
• Experience building scalable batch and/or streaming data pipelines.
• Deep understanding of Bronze/Silver/Gold medallion architectures.
• Experience with Databricks Lakeflow or comparable orchestration technologies.
• Experience implementing enterprise data quality, lineage, observability, and monitoring.
• Experience with Unity Catalog and governed enterprise data environments.
• Strong understanding of dimensional modeling, data products, semantic models, and cross-domain integration.
• Experience with Git-based development, CI/CD, automated testing, and production deployment.
• Ability to troubleshoot complex distributed data and pipeline issues.
• Experience building AI and GenAI solutions against enterprise data.
• Understanding of semantic layers and natural-language-to-data architectures.
• Experience with Databricks Genie or comparable natural-language analytics technology.
• Experience building tools, functions, APIs, or services consumed by AI agents.
• Understanding of MCP, agent tool calling, tracing, evaluation, and AI governance.
• Experience developing production AI solutions with security, observability, and governance controls.
• Preferred: Databricks Data Engineer Professional or equivalent experience.
• Preferred: Experience with Databricks AI/BI, Genie, MLflow, and MCP-based integrations.
• Preferred: Experience in Real Estate, Finance, People/Workplace, Operations, or similar domains.
• Preferred: Experience building metadata-driven or configuration-driven ingestion frameworks.

Responsibilities

• Design and build production-grade data pipelines on Databricks.
• Ingest data from enterprise applications, databases, APIs, SaaS platforms, files, and other source systems.
• Build and maintain Bronze, Silver, and Gold medallion architectures.
• Implement incremental processing and change data capture patterns where appropriate.
• Standardize types, codes, units, keys, and business definitions across source systems.
• Implement deduplication, historical tracking, schema evolution, and data lineage.
• Optimize pipelines for reliability, performance, scalability, and cost.
• Build a configuration-driven source onboarding framework and Lakeflow pipeline templates.
• Automate source registration, deployment, monitoring, lineage, and operational controls.
• Implement data quality expectations, anomaly detection, monitoring, and quarantine processes.
• Implement job and pipeline monitoring, alerting, audit logging, and operational dashboards.
• Support automated deployment and promotion using Databricks Asset Bundles and CI/CD.
• Develop conformed dimensions and shared-key strategies across domains.
• Build integrated Gold-layer products and reusable data contracts.
• Build AI-ready Gold data products with ownership, lineage, definitions, and quality information.
• Develop semantic context, business glossary, ontology, and certified KPIs in Unity Catalog.
• Build governed Unity Catalog functions and agent-ready tools and endpoints.
• Implement tracing, evaluation, testing, monitoring, and governance for AI and agent workflows.
• Support development of AI agents that use trusted enterprise data.

Skills

Databricks

Apache Spark

Delta Lake

Python

SQL

AWS EMR

Bedrock

Unity Catalog

Git CI/CD

GenAI

Tools

Databricks

Apache Spark

Delta Lake

Unity Catalog

Git

CI/CD

MLflow

Databricks Genie

Job description

Eliassen Group is seeking a Staff AI & Data Engineer to design and build enterprise-grade data and AI solutions on Databricks. This hands-on role focuses on strengthening the Databricks platform, onboarding new data sources, and creating scalable ingestion and transformation frameworks.

The position is remote for a U.S.-based client under a W2 contract. You will work across data pipelines, semantic layers, and AI agent-ready capabilities, delivering Gold-layer data products and governance.