Senior Azure Databricks Engineer - Remote, Lakehouse Expert

Dataflix Inc.; United States; A distancia; USD 130.000 - 170.000; Jornada completa

Descripción de la vacante

Dataflix Inc. is seeking a Senior Databricks Engineer to design, build, and optimize scalable data solutions on Azure Databricks.

The ideal candidate will have strong hands-on experience with PySpark, Spark SQL, Delta Lake, Lakehouse architecture, data pipelines, and performance optimization. Responsibilities include designing scalable data pipelines, building Lakehouse architectures, and developing production-grade ETL/ELT processes while ensuring data quality, security, and governance.

Formación

• 7+ years of overall data engineering/software engineering experience in large-scale data platforms.

Responsabilidades

• Design and develop scalable data pipelines using Azure Databricks, Apache Spark, PySpark, Python, and Spark SQL.
• Build and maintain Lakehouse architectures using Delta Lake, Delta Tables, Unity Catalog, and Medallion Architecture (Bronze/Silver/Gold).
• Develop reusable batch and streaming ETL/ELT pipelines using Databricks Workflows, Lakeflow/Delta Live Tables, and Structured Streaming.
• Optimize Spark workloads through partitioning, caching, query optimization, join/shuffle optimization, and performance tuning.
• Develop complex SQL transformations, data models, schemas, and curated datasets for analytics and downstream applications.
• Implement data quality, security, access controls, lineage, monitoring, and error-handling capabilities.
• Build production-ready solutions following engineering best practices including Git, CI/CD, automated testing, code reviews, and deployment automation.
• Collaborate with architects, analysts, data scientists, and business teams to deliver reliable enterprise data solutions.

Conocimientos

Python

PySpark

Spark SQL

Apache Spark

SQL

Delta Lake

Unity Catalog

Medallion Architecture

Databricks Workflows

Structured Streaming

Git

CI/CD

Automated testing

Agile

Educación

Bachelor’s degree in Computer Science, Information Technology, Engineering, Data Science, or related field

Herramientas

Git

CI/CD

Automated testing

Agile engineering practices

Descripción del empleo

Dataflix Inc. is seeking a Senior Databricks Engineer to design, build, and optimize scalable data solutions on Azure Databricks.

The ideal candidate will have strong hands-on experience with PySpark, Spark SQL, Delta Lake, Lakehouse architecture, data pipelines, and performance optimization. Responsibilities include designing scalable data pipelines, building Lakehouse architectures, and developing production-grade ETL/ELT processes while ensuring data quality, security, and governance.