Senior Databricks Engineer - Azure Data Pipelines

Accelyst, LLC; Louisville (KY); On-site; USD 120,000 - 180,000; Full time

Job summary

Accelyst is seeking a Senior Databricks Engineer to design, develop, and optimize scalable data engineering solutions on Azure Databricks. You will build robust data pipelines, support enterprise analytics, and drive data platform initiatives.

The ideal candidate brings 6+ years in data engineering, 3+ years with Azure Databricks, and strong PySpark, Spark SQL, and Delta Lake expertise. Collaboration with analytics teams and a focus on data governance are essential.

Qualifications

• 6+ years of experience in Data Engineering.
• 3+ years hands-on with Azure Databricks.
• Strong expertise in PySpark, Spark SQL, and Delta Lake.
• Experience with Azure Data Factory and ADLS.
• Strong SQL and Python proficiency.
• Knowledge of data warehousing concepts and dimensional modeling.
• Experience with Git-based version control and CI/CD.

Responsibilities

• Design, develop, and maintain scalable data pipelines using Azure Databricks.
• Develop ETL/ELT workflows for large volumes of structured and unstructured data.
• Work with ADLS Gen2, Azure Blob Storage, and other Azure data services.
• Develop data transformations using PySpark, Spark SQL, and Databricks notebooks.
• Optimize Spark jobs for performance, scalability, and cost efficiency.
• Build data ingestion frameworks from multiple source systems.
• Implement CI/CD pipelines and deployment automation for Databricks workloads.
• Collaborate with stakeholders, data architects, and analytics teams to deliver data solutions.
• Ensure data quality, governance, security, and compliance standards.
• Troubleshoot production issues and provide performance-tuning recommendations.

Skills

Databricks

Azure Databricks

PySpark

Spark SQL

Delta Lake

Python

SQL

Data Pipelines

CI/CD

DevOps

Data Warehousing

Education

Bachelor's degree in Computer Science

Tools

Azure Data Factory

ADLS Gen2

Azure Blob Storage

Git

Job description

Accelyst is seeking a Senior Databricks Engineer to design, develop, and optimize scalable data engineering solutions on Azure Databricks. You will build robust data pipelines, support enterprise analytics, and drive data platform initiatives.

The ideal candidate brings 6+ years in data engineering, 3+ years with Azure Databricks, and strong PySpark, Spark SQL, and Delta Lake expertise. Collaboration with analytics teams and a focus on data governance are essential.