Senior AI & Data Engineer - Databricks (Remote)

Eliassen Group; Phoenix (AZ); Remote; USD 110,000 - 124,000; Full time

Benefits offered by this job

Medical benefits

Dental benefits

Vision benefits

401k with company match

Life insurance

Job summary

Eliassen Group is seeking a Staff AI & Data Engineer to design and build enterprise-grade data and AI solutions on Databricks. This remote contract role focuses on strengthening the Databricks platform, onboarding new data sources, and creating trusted Gold-layer data products with AI and GenAI capabilities.

The ideal candidate brings advanced Databricks, Spark, Delta Lake, and Python skills, plus experience with end-to-end data pipelines, governance, and CI/CD for production deployments.

Qualifications

• Advanced hands-on experience with Databricks, Apache Spark, Delta Lake, and SQL.
• Hands-on experience with AWS services related to data analytics and AI, including EMR and Bedrock.
• Strong Python and SQL development skills.
• Experience building scalable batch and/or streaming data pipelines.

Responsibilities

• Design and build production-grade data pipelines on Databricks.
• Ingest data from enterprise applications, databases, APIs, SaaS platforms, files, and other source systems.
• Build and maintain Bronze, Silver, and Gold medallion architectures.
• Implement incremental processing and change data capture patterns where appropriate.
• Standardize types, codes, units, keys, and business definitions across source systems.
• Implement deduplication, historical tracking, schema evolution, and data lineage.
• Optimize pipelines for reliability, performance, scalability, and cost.
• Build a configuration-driven source onboarding framework and Lakeflow pipeline templates.
• Automate source registration, deployment, monitoring, lineage, and operational controls.
• Implement data quality expectations, anomaly detection, monitoring, and quarantine processes.
• Implement job and pipeline monitoring, alerting, audit logging, and operational dashboards.
• Support automated deployment and promotion using Databricks Asset Bundles and CI/CD.
• Develop conformed dimensions and shared-key strategies across domains.
• Build integrated Gold-layer products and reusable data contracts.
• Build AI-ready Gold data products with ownership, lineage, definitions, and quality information.
• Develop semantic context, business glossary, ontology, and certified KPIs in Unity Catalog.
• Build governed Unity Catalog functions and agent-ready tools and endpoints.
• Implement tracing, evaluation, testing, monitoring, and governance for AI and agent workflows.
• Support development of AI agents that use trusted enterprise data.

Skills

Databricks

Apache Spark

Delta Lake

SQL

Python

AWS

Data pipelines

Data governance

Tools

Databricks Lakeflow

Job description

Eliassen Group is seeking a Staff AI & Data Engineer to design and build enterprise-grade data and AI solutions on Databricks. This remote contract role focuses on strengthening the Databricks platform, onboarding new data sources, and creating trusted Gold-layer data products with AI and GenAI capabilities.

The ideal candidate brings advanced Databricks, Spark, Delta Lake, and Python skills, plus experience with end-to-end data pipelines, governance, and CI/CD for production deployments.