Senior Databricks Engineer - REMOTE

Sligo Software Solutions Inc.; City of Albany (NY); On-site; USD 150,000 - 195,000; Full time

Job summary

Sligo Software Solutions Inc. in New York, NY seeks an experienced Data Engineer to lead modern data pipelines and power analytics across platforms. You will implement scalable ETL/ELT processes using Databricks, Spark, PySpark, and Python, ensuring robust data models and governance.

Ideal candidates have 7+ years in data engineering, strong SQL, hands-on Delta Lake/Lakehouse experience, and cloud Databricks exposure (Azure/AWS). Healthcare domain knowledge is a plus; hands-on contributors only.

Qualifications

• 7+ years of data engineering experience.
• Hands-on Databricks, Spark, PySpark, and Python.
• Strong SQL and data engineering capabilities.
• Experience building scalable ETL/ELT pipelines with Databricks.
• Familiarity with Delta Lake and Lakehouse architecture.
• Experience with cloud Databricks (Azure/AWS).
• Hands-on with Databricks Workflows, Jobs, notebooks, and performance tuning.
• Experience integrating data from databases, APIs, files, and cloud sources.
• CI/CD, Git and DevOps practices in data projects.
• Data modeling, data quality, and governance.
• Healthcare/Health insurance domain experience is highly preferred.
• Must be hands-on and not primarily production-support/management.

Skills

Data engineering

Databricks

Apache Spark

PySpark

Python

SQL

ETL/ELT pipelines

Delta Lake

Lakehouse

CI/CD

Git

DevOps practices

Data modeling

Data quality

Data governance

Healthcare domain

Tools

Databricks Platform

Delta Lake

Azure Databricks

AWS Databricks

Databricks Workflows

Notebooks

Job description

• 7+ years of experience in Data Engineering, with strong recent hands-on Databricks experience.
• Strong hands-on experience with Databricks, Apache Spark, PySpark, and Python.
• Strong SQL and data engineering experience.
• Experience building scalable ETL/ELT data pipelines using Databricks.
• Strong experience with Delta Lake and Databricks Lakehouse architecture.
• Experience with Azure Databricks or AWS Databricks.
• Experience with Databricks Workflows, Jobs, notebooks, and performance optimization.
• Experience integrating data from databases, APIs, files, and cloud data sources.
• Experience with CI/CD, Git, and DevOps practices.
• Experience with data modeling, data quality, and data governance.
• Healthcare / Health Insurance experience is highly preferred.
• Must be hands-on development focused; avoid candidates whose background is primarily production support or management.