Databricks Engineer

Dataflix; United States; A distancia; USD 130.000 - 170.000; Jornada completa

Descripción de la vacante

Dataflix Inc. is seeking a Senior Databricks Engineer to design, build, and optimize scalable data solutions on Azure Databricks.

The ideal candidate will have strong hands-on experience with PySpark, Spark SQL, Delta Lake, Lakehouse architecture, data pipelines, and performance optimization. Responsibilities include designing scalable data pipelines, building Lakehouse architectures, and developing production-grade ETL/ELT processes while ensuring data quality, security, and governance.

Formación

• 7+ years of overall data engineering/software engineering experience in large-scale data platforms.

Responsabilidades

• Design and develop scalable data pipelines using Azure Databricks, Apache Spark, PySpark, Python, and Spark SQL.
• Build and maintain Lakehouse architectures using Delta Lake, Delta Tables, Unity Catalog, and Medallion Architecture (Bronze/Silver/Gold).
• Develop reusable batch and streaming ETL/ELT pipelines using Databricks Workflows, Lakeflow/Delta Live Tables, and Structured Streaming.
• Optimize Spark workloads through partitioning, caching, query optimization, join/shuffle optimization, and performance tuning.
• Develop complex SQL transformations, data models, schemas, and curated datasets for analytics and downstream applications.
• Implement data quality, security, access controls, lineage, monitoring, and error-handling capabilities.
• Build production-ready solutions following engineering best practices including Git, CI/CD, automated testing, code reviews, and deployment automation.
• Collaborate with architects, analysts, data scientists, and business teams to deliver reliable enterprise data solutions.

Conocimientos

Python

PySpark

Spark SQL

Apache Spark

SQL

Delta Lake

Unity Catalog

Medallion Architecture

Databricks Workflows

Structured Streaming

Git

CI/CD

Automated testing

Agile

Educación

Bachelor’s degree in Computer Science, Information Technology, Engineering, Data Science, or related field

Herramientas

Git

CI/CD

Automated testing

Agile engineering practices

Descripción del empleo

We design, build, and run modern data platforms and AI-native systems that drive productivity, efficiency, and quality — while enabling businesses to move faster and adapt to change.

Job Description

This is a remote position.

Dataflix is seeking a Senior Databricks Engineer to design, build, and optimize scalable data solutions on Azure Databricks. The ideal candidate will have strong hands-on experience with PySpark, Spark SQL, Delta Lake, Lakehouse architecture, data pipelines, and performance optimization.

Responsibilities

• Design and develop scalable data pipelines using Azure Databricks, Apache Spark, PySpark, Python, and Spark SQL.
  
• Build and maintain Lakehouse architectures using Delta Lake, Delta Tables, Unity Catalog, and Medallion Architecture (Bronze/Silver/Gold).
  
• Develop reusable batch and streaming ETL/ELT pipelines using Databricks Workflows, Lakeflow/Delta Live Tables, and Structured Streaming.
  
• Optimize Spark workloads through partitioning, caching, query optimization, join/shuffle optimization, and performance tuning.
  
• Develop complex SQL transformations, data models, schemas, and curated datasets for analytics and downstream applications.
  
• Implement data quality, security, access controls, lineage, monitoring, and error-handling capabilities.
  
• Build production-ready solutions following engineering best practices including Git, CI/CD, automated testing, code reviews, and deployment automation.
  
• Collaborate with architects, analysts, data scientists, and business teams to deliver reliable enterprise data solutions.
  

Required Skills

Bachelor’s degree in Computer Science, Information Technology, Engineering, Data Science, or a related technical field. 7+ years of overall data engineering/software engineering experience, with significant hands-on experience in large-scale data platforms.

3+ years of hands-on Azure Databricks experience , including production implementations.

• Strong hands-on expertise in Python, PySpark, Spark SQL, Apache Spark, and SQL.
  
• Proven experience with Delta Lake, Unity Catalog, Medallion Architecture, Databricks Workflows/Lakeflow, and Structured Streaming.
  
• Experience designing and supporting production-grade ETL/ELT pipelines processing large volumes of data.
  
• Strong understanding of data modeling, performance optimization, data quality, security, governance, and monitoring.
  
• Experience with Git, CI/CD, automated testing, and Agile engineering practices.
  
• Databricks certifications such as Databricks Certified Data Engineer Associate/Professional are preferred but not required.
  

Preferred

• Experience with ADLS Gen2, Azure Data Factory, Azure SQL, or other Azure data services
  
• Databricks certification
• Experience with infrastructure/deployment automation and enterprise data governance