Azure Databricks Data Engineer - Spark Pipelines & Lakehouse

Fractal Analytics; California (MO); On-site; USD 120,000 - 140,000; Full time

Benefits offered by this job

Health insurance

401(k) plan

Paid holidays

Parental leave

PTO policy

Job summary

Fractal Analytics is seeking an experienced Azure Databricks Data Engineer to design, develop, and maintain scalable cloud-based data solutions. The role requires deep expertise in Azure data engineering tools, Databricks, data pipelines, data warehousing, and big data processing.

The candidate will collaborate with architects, data scientists, and application teams to deliver data platforms supporting analytics and AI initiatives, while ensuring security, governance, and scalability across

Qualifications

• Bachelor's or master's degree in CS/IT/Engineering.
• 6-10 years in Data Engineering and Analytics.
• Minimum 3+ years hands-on Azure Databricks experience.
• Strong PySpark and SQL, cloud-native data platforms.
• Experience building data lakes and data warehouses.
• Knowledge of Spark performance tuning.
• Experience in Agile delivery.

Responsibilities

• Design, build, and optimize scalable data pipelines with Azure Databricks.
• Develop ETL/ELT workflows for large datasets.
• Implement batch and real-time processing using PySpark and Spark SQL.
• Create data ingestion frameworks from multiple sources.
• Develop data lakes with ADLS Gen2 and manage data schemas.
• Integrate data from cloud and on-prem systems.
• Ensure data quality, monitoring, and governance.
• Optimize Spark jobs for cost and performance.
• Collaborate with stakeholders to define reporting needs.
• Automate deployments via CI/CD and participate in architecture reviews.

Skills

Azure Databricks

PySpark

Spark SQL

Delta Lake

ETL/ELT

Data Warehousing

Python

SQL

Azure Data Lake Storage Gen2

Azure Synapse Analytics

Azure Data Factory

Azure Monitor

Azure DevOps

CI/CD Pipelines

Terraform (preferred)

Data Modeling

Data Governance

Python

Scala (Preferred)

Education

Bachelor's or Master's in Computer Science/IT/Engineering

Tools

Azure Databricks

Azure Data Factory

ADLS Gen2

Azure Synapse

Azure DevOps

Terraform

Git

CI/CD tooling

Job description

Fractal Analytics is seeking an experienced Azure Databricks Data Engineer to design, develop, and maintain scalable cloud-based data solutions. The role requires deep expertise in Azure data engineering tools, Databricks, data pipelines, data warehousing, and big data processing.

The candidate will collaborate with architects, data scientists, and application teams to deliver data platforms supporting analytics and AI initiatives, while ensuring security, governance, and scalability across