Senior Databricks Engineer (Onsite Interview) (ONLY W2)

Accelyst, LLC; Louisville (KY); On-site; USD 120,000 - 180,000; Full time

Job summary

Accelyst is seeking a Senior Databricks Engineer to design, develop, and optimize scalable data engineering solutions on Azure Databricks. You will build robust data pipelines, support enterprise analytics, and drive data platform initiatives.

The ideal candidate brings 6+ years in data engineering, 3+ years with Azure Databricks, and strong PySpark, Spark SQL, and Delta Lake expertise. Collaboration with analytics teams and a focus on data governance are essential.

Qualifications

• 6+ years of experience in Data Engineering.
• 3+ years hands-on with Azure Databricks.
• Strong expertise in PySpark, Spark SQL, and Delta Lake.
• Experience with Azure Data Factory and ADLS.
• Strong SQL and Python proficiency.
• Knowledge of data warehousing concepts and dimensional modeling.
• Experience with Git-based version control and CI/CD.

Responsibilities

• Design, develop, and maintain scalable data pipelines using Azure Databricks.
• Develop ETL/ELT workflows for large volumes of structured and unstructured data.
• Work with ADLS Gen2, Azure Blob Storage, and other Azure data services.
• Develop data transformations using PySpark, Spark SQL, and Databricks notebooks.
• Optimize Spark jobs for performance, scalability, and cost efficiency.
• Build data ingestion frameworks from multiple source systems.
• Implement CI/CD pipelines and deployment automation for Databricks workloads.
• Collaborate with stakeholders, data architects, and analytics teams to deliver data solutions.
• Ensure data quality, governance, security, and compliance standards.
• Troubleshoot production issues and provide performance-tuning recommendations.

Skills

Databricks

Azure Databricks

PySpark

Spark SQL

Delta Lake

Python

SQL

Data Pipelines

CI/CD

DevOps

Data Warehousing

Education

Bachelor's degree in Computer Science

Tools

Azure Data Factory

ADLS Gen2

Azure Blob Storage

Git

Job description

Accelyst is an innovative AI Consultancy that leverages industry-specific AI Agents and leading-edge AI platforms to deliver integrated, secure, and ROI-optimized solutions. We combine deep industry and technical expertise to enable rapid deployment of AI-driven capabilities that augment and automate client workflows.

Why Accelyst?

Join Accelyst to work in a dynamic, client-focused environment on complex data and AI initiatives. You’ll collaborate with experienced technology teams, work with modern cloud platforms, and contribute to scalable data solutions that support enterprise analytics and digital transformation.

Job Summary

Accelyst is seeking a highly skilled Senior Databricks Engineer with strong hands-on experience in Azure Cloud and Azure Databricks. The ideal candidate will design, develop, and optimize scalable data engineering solutions, build robust data pipelines, and support enterprise analytics and data platform initiatives.

Job Roles and Responsibilities

• Design, develop, and maintain scalable data pipelines using Azure Databricks.
• Develop ETL/ELT workflows for large volumes of structured and unstructured data.
• Work with ADLS Gen2, Azure Blob Storage, and other Azure data services.
• Develop data transformations using PySpark, Spark SQL, and Databricks notebooks.
• Optimize Spark jobs for performance, scalability, and cost efficiency.
• Build data ingestion frameworks from multiple source systems.
• Implement CI/CD pipelines and deployment automation for Databricks workloads.
• Collaborate with stakeholders, data architects, and analytics teams to deliver data solutions.
• Ensure data quality, governance, security, and compliance standards.
• Troubleshoot production issues and provide performance-tuning recommendations.

Job Requirement Qualifications

• 6+ years of experience in Data Engineering.
• 3+ years of hands-on experience with Azure Databricks.
• Strong expertise in PySpark, Spark SQL, and Delta Lake.
• Experience with Azure Data Factory (ADF) and Azure Data Lake Storage (ADLS).
• Strong proficiency in SQL and Python.
• Experience with data warehousing concepts and dimensional modeling.
• Experience with Git-based version control and CI/CD processes.
• Understanding of DevOps practices and deployment automation.
• Strong analytical and problem-solving skills.

Preferred Skills

• Experience with Delta Live Tables (DLT).
• Knowledge of Unity Catalog and Databricks governance frameworks.
• Exposure to Azure Synapse Analytics.
• Experience with streaming technologies and real-time data processing.
• Databricks or Azure Cloud certifications.

Accelyst is an equal employment opportunity employer. All qualified applicants will receive consideration for employment without regard to any characteristic protected by applicable law.