Databricks Lakehouse Architect - Real-Time Data & ML

Polestar Analytics; United States; On-site; USD 120,000 - 160,000; Full time

Job summary

Polestar Solutions, a data analytics and enterprise planning powerhouse, seeks a senior data platform engineer to design and implement modern data platforms using Databricks Lakehouse architecture. You will define data architectures, models, and governance for enterprise analytics, and build scalable batch and real-time data pipelines.

You will develop and optimize ETL/ELT pipelines with Apache Spark, Python, and SQL, and implement ingestion frameworks using Azure Data Factory, Kafka, or

Qualifications

• Strong experience with Databricks platform.
• Expertise in Apache Spark, Spark SQL, and distributed computing.
• Programming experience in Python, Scala, or SQL.
• Experience with data lakes and lakehouse architectures.
• Strong experience with cloud data platforms.

Responsibilities

• Design and implement modern data platforms using Databricks Lakehouse architecture.
• Define data architecture, data models, and governance for enterprise analytics platforms.
• Architect scalable batch and real-time data pipelines.
• Develop and optimize ETL/ELT pipelines using Apache Spark, Python, and SQL.
• Implement data ingestion frameworks using Azure Data Factory, Apache Kafka, or Apache Airflow.
• Work with structured, semi-structured, and unstructured data.
• Optimize Spark workloads, cluster configurations, and query performance.
• Implement data quality, monitoring, and cost optimization strategies.
• Ensure data platform scalability, reliability, and security.
• Enable ML pipelines and collaboration with data science teams using MLflow.
• Build feature engineering pipelines and support AI/ML use cases.

Skills

Databricks platform

Apache Spark

Spark SQL

Python

Scala

SQL

Lakehouse architecture

Cloud data platforms

Tools

MLflow

Azure Data Factory

Apache Kafka

Apache Airflow

Job description

Polestar Solutions, a data analytics and enterprise planning powerhouse, seeks a senior data platform engineer to design and implement modern data platforms using Databricks Lakehouse architecture. You will define data architectures, models, and governance for enterprise analytics, and build scalable batch and real-time data pipelines.

You will develop and optimize ETL/ELT pipelines with Apache Spark, Python, and SQL, and implement ingestion frameworks using Azure Data Factory, Kafka, or