Senior Data Engineer

PharmaLink, LLC; Largo (FL); On-site; USD 120,000 - 180,000; Full time

Job summary

PharmaLink, LLC is seeking a Senior Data Engineer to design, build, and operate an enterprise data and AI platform in Largo, FL. You will focus on the Databricks ecosystem, develop Python-based ETL/ELT pipelines, and implement data governance with Unity Catalog.

The role emphasizes scalable data products, AI enablement, and cost optimization. The ideal candidate has extensive Databricks experience, strong SQL/PySpark skills, and a track record of building lakehouse architectures in cloud

Qualifications

• 8+ years in data engineering or related roles.
• 4+ years hands-on Databricks experience in production.
• Advanced SQL, Python, PySpark, Spark SQL.
• Strong Delta Lake and Unity Catalog experience.
• ETL/ELT, data modeling, and enterprise data architecture.
• Experience with Git, CI/CD, testing, monitoring, and production support.
• Experience with cloud-based enterprise data platforms.

Responsibilities

• Design, build, and operate enterprise data solutions on Databricks.
• Develop scalable ETL/ELT pipelines in Python, PySpark, and Spark SQL.
• Create curated data models and reusable data products for analytics and AI.
• Implement Unity Catalog, security, lineage, and data governance.
• Build and optimize Databricks SQL Warehouses and AI/BI dashboards.
• Develop data/AI apps using Databricks Apps and Lakebase.
• Enable AI/ML and Generative AI use cases with Databricks features.
• Automate data quality, reconciliation, anomaly detection, and monitoring.
• Integrate data from databases, SaaS, APIs, and files.
• Optimize Spark workloads, queries, pipelines, and costs.
• Establish engineering standards for Git, CI/CD, and testing.
• Troubleshoot production/data issues and perform root cause analysis.
• Collaborate with governance, BI, apps, and business teams.

Skills

Databricks

SQL

Python

PySpark

Spark SQL

Delta Lake

Unity Catalog

ETL/ELT

Data modeling

Enterprise data architecture

Git

CI/CD

Monitoring

Production support

Cloud platforms

Tools

Azure Data Factory

Microsoft SQL Server

Fivetran

Power BI

Databricks Workflows

MLflow

Job description

The Senior Data Engineer works onsite in Largo, FL and is responsible for building and evolving PharmaLink’s enterprise data, analytics, and AI platform. The position focuses on the modern Databricks ecosystem, covering lakehouse architecture, analytics enablement, AI development, and data governance.

Key Responsibilities

• Design, build, and operate enterprise data solutions on Databricks.
• Develop scalable ETL and ELT pipelines using Python, PySpark, Spark SQL, and Delta Lake.
• Create curated data models and reusable data products to support analytics, applications, and AI.
• Implement Unity Catalog and related security, lineage, access controls, and data-governance capabilities.
• Build and optimize Databricks SQL Warehouses and AI/BI Dashboards.
• Develop data and AI applications using Databricks Apps.
• Design and implement solutions using Databricks Lakebase.
• Enable AI/ML and Generative AI use cases using Databricks capabilities.
• Build automated data-quality, reconciliation, anomaly-detection, and monitoring processes.
• Integrate data from databases, SaaS platforms, APIs, files, and other enterprise systems.
• Optimize Spark workloads, SQL queries, pipelines, performance, and platform cost.
• Establish engineering standards for Git, CI/CD, testing, monitoring, and production deployment.
• Troubleshoot complex production and data issues and perform root-cause analysis.
• Partner with Data Governance, BI, application, and business teams to deliver trusted data products.

Required Qualifications

• 8+ years of experience in data engineering, data platforms, or related engineering roles.
• 4+ years of hands-on Databricks experience in production environments.
• Advanced SQL, Python, PySpark, and Apache Spark skills.
• Strong experience with Delta Lake and Unity Catalog.
• Strong ETL/ELT, data modeling, and enterprise data architecture experience.Experience with Databricks SQL and AI/BI Dashboards.
• Familiarity with Databricks capabilities across AI/ML, Generative AI, Databricks Apps, and Lakebase.
• Strong relational database and data-engineering fundamentals.
• Experience with Git, CI/CD, automated testing, monitoring, and production support.
• Experience working with cloud-based enterprise data platforms.
• Must maintain a positive attitude in a high-performance team environment.
• Must pass background check and random drug screenings.
• Must be adaptable and flexible in dealing with a diverse work force.

Preferred Qualifications

• Databricks on Microsoft Azure.
• Azure Data Factory.
• Microsoft SQL Server.
• Fivetran or similar CDC/ELT technologies.
• Power BI.
• Databricks Workflows.
• MLflow and AI/model lifecycle capabilities.
• API and application development experience.
• Azure DevOps or GitHub-based CI/CD.
• Databricks professional certification.
• Experience in healthcare, life sciences, or another regulated industry.

Technologies

• Databricks, Python, PySpark, Spark SQL, Delta Lake, Unity Catalog
• Databricks SQL, AI/BI Dashboards, Databricks Apps, Lakebase
• AI/ML, Generative AI
• Git, CI/CD, Apache Spark

Physical Requirements

Requires light physical activity performing non-strenuous daily activities of an administrative nature (i.e., sustained phone, headset, and keyboard use) as well as periods of prolonged sitting.