Databricks Data Engineer: Build Scalable Lakehouse

Aether Biomedical; United States; Remote; USD 120,000 - 170,000; Full time

Job summary

Aether Biomedical is seeking a Data Engineer with strong Databricks experience to support a data & AI project building a modern, scalable data platform for analytics, ML and AI use cases.

You will design, build and optimize data pipelines, integrate data from multiple sources, and collaborate with data scientists and stakeholders to deliver data products and a high-quality data layer.

Qualifications

• Hands-on Databricks experience required.
• Strong SQL and PySpark/Spark skills.
• Experience with large datasets and distributed processing.
• Knowledge of lakehouse and modern data architecture concepts.
• Cloud experience (Azure/AWS/GCP) is a strong plus.

Responsibilities

• Design, build and maintain scalable data pipelines using Databricks and Spark.
• Integrate data from multiple sources and build reliable, reusable data flows.
• Develop and optimize data processing solutions for large volumes of data.
• Collaborate with Data Scientists, Analysts and stakeholders to deliver data products.
• Contribute to the development of a unified data platform and high-quality data layer.
• Ensure data pipelines are reliable, scalable, performant and easy to maintain.
• Optimize data processing and infrastructure with a focus on performance, scalability and cost efficiency.
• Implement and maintain data quality, monitoring and data engineering best practices.
• Support the development and evolution of modern lakehouse and cloud data architectures.

Skills

Databricks

Data Engineering

PySpark / Spark

SQL

Large datasets

Data architecture

Cloud platforms (Azure/AWS/GCP)

Delta Lake

Data orchestration tools

Data sources integration

Tools

Apache Spark

Job description

Aether Biomedical is seeking a Data Engineer with strong Databricks experience to support a data & AI project building a modern, scalable data platform for analytics, ML and AI use cases.

You will design, build and optimize data pipelines, integrate data from multiple sources, and collaborate with data scientists and stakeholders to deliver data products and a high-quality data layer.