Data Engineer: Spark, Databricks & Lakehouse Architect

Quantum Technologies. LLC

Houston (TX)

Vor Ort

USD 110.000 - 170.000

Vollzeit

vor 4 Stunden

Sei unter den ersten Bewerbenden

Erhalte eine Antwort von diesem Recruiter

Dein Lebenslauf und dein Anschreiben, genau auf das zugeschnitten, was gesucht wird.

Schaffe es an den ATS-Filtern vorbei

Zusammenfassung

Quantum Technologies. LLC in Houston is seeking a seasoned Databricks Engineer to design, develop, and optimize Spark-based data pipelines.

You will work with Spark DataFrames, Spark SQL, PySpark, and SQL to process large volumes of data in Azure Databricks, ensuring robust performance and scalable lakehouse solutions. You will monitor live Databricks jobs, troubleshoot issues, and collaborate with DevOps to manage Docker/Kubernetes deployments.

Qualifikationen

Develop and optimize Spark jobs, data transformations, and processing workflows.
Implement complex data streaming and transformation solutions using Azure Databricks.
Design and implement data warehouse/lakehouse solutions.

Aufgaben

Develop and optimize Spark jobs, data transformations, and processing workflows.
Implement complex data streaming and transformation solutions using Azure Databricks.
Design and implement data warehouse/lakehouse solutions.
Monitor, debug, and resolve issues in live Databricks jobs.
Troubleshoot DevOps pipelines, Azure, and AKS services.
Build data ingestion, streaming, and transformation solutions.

Kenntnisse

Apache Spark

Spark DataFrames

Spark SQL

PySpark

SQL

Azure Databricks

Databricks Job Orchestration

Databricks Serverless

Streaming

ETL/ELT

Data Warehousing

Kubernetes

Docker

Scala

Java

Tools