Lead Data Engineer, Azure Databricks & AI Pipelines

Alimentiv-2; San Diego (CA); On-site; USD 150,000 - 210,000; Full time

Benefits offered by this job

Education Reimbursement program for 1

Monthly Cell Phone and Remote Work 4

Office setup allowance

Visiting program – up to 4 weeks/yr

Flexible Summer Hours

Employee referral incentive

Alimentiv Trust after 1 year / 5 years

Job summary

Alimentiv is seeking a Lead Data Engineer to design and deliver scalable data solutions supporting analytics, BI, and AI/ML. You will lead modern data architectures and automated pipelines using Azure Databricks, Fabric, PySpark, SQL, and related Azure tech.

You will mentor engineers, drive data quality and governance, and collaborate with stakeholders to translate business needs into robust data solutions across the enterprise.

Qualifications

• Bachelor's degree or higher in a related field; Master’s preferred.
• 5–8 years of experience delivering enterprise-scale data solutions, including data warehouses and data lakes.
• Expert-level proficiency in Azure Databricks, Microsoft Fabric, PySpark, SQL, and Azure DevOps.
• Hands-on experience with Azure Data Factory, ADLS Gen2, and Azure SQL Server.
• Understanding of modern data architectures, data governance, metadata management, and data cataloguing.
• Experience with REST APIs, JSON, real-time ingestion, and reusable integration frameworks.
• CI/CD practices and infrastructure-as-code (ARM/Bicep/Terraform).
• Experience with Power BI and BI data modeling; Tableau/Looker a plus.
• Regulatory knowledge: GxP, GDPR, HIPAA in life sciences domain is desirable.

Responsibilities

• Architect and implement scalable end-to-end data solutions using Azure Databricks, Fabric, PySpark, and Azure SQL.
• Develop and optimise data pipelines for ingestion, transformation, and consumption across platforms.
• Design data models across bronze, silver, and gold layers in data lake/warehouse architectures.
• Improve Spark/SQL performance and cost efficiency.
• Build API integrations, real-time ingestion, and reusable templates.
• Lead CI/CD automation, tests, and production deployments.
• Prepare datasets and models for BI, self-service analytics, and AI/ML initiatives.
• Ensure data governance, security, and regulatory compliance.
• Collaborate with stakeholders to translate requirements into data engineering solutions.
• Provide technical leadership and mentor junior engineers.

Skills

Azure Databricks

Microsoft Fabric

PySpark

SQL

Azure DevOps

Azure Data Factory

ADLS Gen2

Azure SQL Server

Power BI

Data modeling

Regulatory governance

REST APIs

JSON

ARM/Bicep/Terraform

Git

Tableau

Looker

Education

Bachelor's degree in Computer Science, Information Systems, Engineering, or related; Master’s preferred

Tools

Power BI

Tableau

Looker

Git

CI/CD tooling (Azure DevOps)

Job description

Alimentiv is seeking a Lead Data Engineer to design and deliver scalable data solutions supporting analytics, BI, and AI/ML. You will lead modern data architectures and automated pipelines using Azure Databricks, Fabric, PySpark, SQL, and related Azure tech.

You will mentor engineers, drive data quality and governance, and collaborate with stakeholders to translate business needs into robust data solutions across the enterprise.