Senior Databricks Data Engineer — AI, Lakehouse & Governance

PharmaLink, LLC; Largo (FL); On-site; USD 120,000 - 180,000; Full time

Job summary

PharmaLink, LLC is seeking a Senior Data Engineer to design, build, and operate an enterprise data and AI platform in Largo, FL. You will focus on the Databricks ecosystem, develop Python-based ETL/ELT pipelines, and implement data governance with Unity Catalog.

The role emphasizes scalable data products, AI enablement, and cost optimization. The ideal candidate has extensive Databricks experience, strong SQL/PySpark skills, and a track record of building lakehouse architectures in cloud

Qualifications

• 8+ years in data engineering or related roles.
• 4+ years hands-on Databricks experience in production.
• Advanced SQL, Python, PySpark, Spark SQL.
• Strong Delta Lake and Unity Catalog experience.
• ETL/ELT, data modeling, and enterprise data architecture.
• Experience with Git, CI/CD, testing, monitoring, and production support.
• Experience with cloud-based enterprise data platforms.

Responsibilities

• Design, build, and operate enterprise data solutions on Databricks.
• Develop scalable ETL/ELT pipelines in Python, PySpark, and Spark SQL.
• Create curated data models and reusable data products for analytics and AI.
• Implement Unity Catalog, security, lineage, and data governance.
• Build and optimize Databricks SQL Warehouses and AI/BI dashboards.
• Develop data/AI apps using Databricks Apps and Lakebase.
• Enable AI/ML and Generative AI use cases with Databricks features.
• Automate data quality, reconciliation, anomaly detection, and monitoring.
• Integrate data from databases, SaaS, APIs, and files.
• Optimize Spark workloads, queries, pipelines, and costs.
• Establish engineering standards for Git, CI/CD, and testing.
• Troubleshoot production/data issues and perform root cause analysis.
• Collaborate with governance, BI, apps, and business teams.

Skills

Databricks

SQL

Python

PySpark

Spark SQL

Delta Lake

Unity Catalog

ETL/ELT

Data modeling

Enterprise data architecture

Git

CI/CD

Monitoring

Production support

Cloud platforms

Tools

Azure Data Factory

Microsoft SQL Server

Fivetran

Power BI

Databricks Workflows

MLflow

Job description

PharmaLink, LLC is seeking a Senior Data Engineer to design, build, and operate an enterprise data and AI platform in Largo, FL. You will focus on the Databricks ecosystem, develop Python-based ETL/ELT pipelines, and implement data governance with Unity Catalog.

The role emphasizes scalable data products, AI enablement, and cost optimization. The ideal candidate has extensive Databricks experience, strong SQL/PySpark skills, and a track record of building lakehouse architectures in cloud