Azure Databricks Data Engineer: Scalable Data Pipelines

Fractal Analytics Inc.; California (MO); On-site; USD 120,000 - 140,000; Full time

Benefits offered by this job

Health insurance

Dental insurance

Vision insurance

Life insurance

Disability insurance

401(k) Plan

11 paid holidays

Parental Leave

Job summary

Fractal Analytics Inc. is seeking an experienced Azure Databricks Data Engineer to design, develop, and maintain scalable cloud-based data platforms that support analytics and AI initiatives.

You will work with stakeholders, architects, data scientists, and application teams to deliver high-quality data solutions. The role focuses on building data pipelines, optimizing Spark workloads, and enforcing governance and security.

Qualifications

• 6-10 years of experience in Data Engineering and Analytics.
• Minimum 3+ years of hands-on experience with Azure Databricks.
• Strong experience with PySpark, SQL, and cloud-native data platforms.
• Experience building enterprise-scale data lakes and data warehouses.
• Knowledge of performance tuning and optimization techniques in Spark.
• Experience working in Agile delivery environments.

Responsibilities

• Design, build, and optimize scalable data pipelines using Azure Databricks.
• Develop ETL/ELT workflows for large-scale structured and unstructured datasets.
• Implement batch and real-time data processing using PySpark and Spark SQL.
• Build and maintain data ingestion frameworks from multiple data sources.
• Develop and manage data lake solutions using ADLS Gen2.
• Integrate data from cloud and on-premises systems into enterprise data platforms.
• Implement data quality, validation, and monitoring frameworks.
• Optimize Spark jobs for cost, scalability, and performance.
• Design dimensional models and support data warehouse initiatives.
• Collaborate with business users to understand reporting and analytics requirements.
• Automate deployment processes using CI/CD pipelines.
• Participate in architecture discussions and provide technical leadership to junior engineers.
• Ensure adherence to security, governance, and compliance standards.

Skills

PySpark

Spark SQL

Delta Lake

ETL/ELT Pipeline Development

Data Modeling

Data Warehousing Concepts

Data Governance and Data Quality

Python

SQL

Scala (Preferred)

Azure DevOps

Education

Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, or related field

Tools

Azure Databricks

Azure Data Factory

Azure Data Lake Storage Gen2

Azure Synapse Analytics

Azure Key Vault

Azure Monitor

Azure DevOps

Job description

Fractal Analytics Inc. is seeking an experienced Azure Databricks Data Engineer to design, develop, and maintain scalable cloud-based data platforms that support analytics and AI initiatives.

You will work with stakeholders, architects, data scientists, and application teams to deliver high-quality data solutions. The role focuses on building data pipelines, optimizing Spark workloads, and enforcing governance and security.