Databricks Engineer

StaffXpert LLC; Birmingham (AL); Hybrid; USD 120,000 - 180,000; Full time

Job summary

STAFFXPERT LLC is seeking a Databricks Engineer to design, develop, and optimize enterprise-grade data solutions on the Databricks platform. The role emphasizes PySpark, SQL, Delta Lake, and data quality frameworks, with production-grade delivery.

Remote/Hybrid arrangement supported; candidate will collaborate with cross-functional teams to modernize data workflows and migrate from legacy ETL tools, ensuring scalable, reliable data pipelines and dashboards using Databricks SQL.

Qualifications

• Bachelor's degree or equivalent experience in CS/IT/Engineering.
• Strong hands-on experience with Databricks, PySpark, SQL, and Delta Lake.
• Proven experience building, optimizing, and maintaining large-scale data pipelines.
• Expertise in data quality, validation, monitoring, and governance frameworks.
• Experience creating dashboards and reporting using Databricks SQL.

Responsibilities

• Design, develop, and optimize Delta Lake data pipelines using PySpark and SQL.
• Implement data transformation processes including cleansing, deduplication, validation and MERGE/upsert operations.
• Build and maintain data quality frameworks, monitoring solutions, and reporting mechanisms.
• Develop Databricks SQL dashboards and AI/BI reporting to track data quality metrics.
• Migrate workflows from legacy ETL tools to Databricks using PySpark, SQL, and Lakeflow Jobs.
• Collaborate with cross-functional teams to deliver high-quality data solutions.
• Perform performance tuning and troubleshooting to improve reliability and efficiency of data pipelines.
• Provide technical guidance and knowledge transfer to internal and client teams.

Skills

Databricks

PySpark

SQL

Delta Lake

Data quality

Data governance

Databricks SQL

Education

Bachelor's degree in Computer Science, Information Technology, Engineering

Tools

Lakeflow Jobs

Lakehouse tooling

Job description

Job Title Databricks Engineer

Location Remote / Hybrid (as applicable)

Job Summary

STAFFXPERT LLC is seeking a Databricks Engineer on behalf of our client in Remote/Hybrid location. This role requires a highly skilled data engineering professional with proven experience designing, developing, and optimizing data solutions on the Databricks platform. The ideal candidate will bring strong expertise in PySpark, SQL, Delta Lake, data quality frameworks, and workflow modernization initiatives, with a track record of delivering enterprise-scale Databricks solutions in production environments.

Key Responsibilities

• Design, develop, and optimize scalable Delta Lake data pipelines using PySpark and SQL.

• Implement data transformation processes including data cleansing, deduplication, validation, and MERGE/upsert operations.

• Build and maintain data quality frameworks, monitoring solutions, and reporting mechanisms.

• Develop Databricks SQL dashboards and AI/BI reporting solutions to track data quality metrics and remediation activities.

• Migrate existing workflows from legacy ETL tools to Databricks using PySpark, SQL, and Lakeflow Jobs.

• Collaborate with cross-functional teams, stakeholders, and developers to deliver high-quality data solutions.

• Perform performance tuning and troubleshooting to improve reliability, scalability, and efficiency of data pipelines.

• Provide technical guidance, best practices, and knowledge transfer to internal and client teams.

Required Qualifications

• Bachelor's degree in Computer Science, Information Technology, Engineering, or a related field, or equivalent experience.

• Strong hands‑on experience with Databricks, PySpark, SQL, and Delta Lake.

• Proven experience building, optimizing, and maintaining large‑scale data processing pipelines.

• Expertise in implementing data quality, validation, monitoring, and governance frameworks.

• Experience creating data visualization, reporting, and dashboard solutions using Databricks SQL.

• Strong understanding of modern data lake and data warehouse architectures.

• Excellent analytical, problem‑solving, and communication skills.

• Demonstrated experience delivering Databricks‑based solutions in production environments.

Preferred Qualifications

• Experience migrating workflows from Alteryx or similar ETL platforms to Databricks.

• Familiarity with Lakehouse architecture and cloud platforms such as AWS, Azure, or GCP.

• Experience mentoring technical teams and leading knowledge‑transfer sessions.

• Databricks certifications are a plus.