Remote Databricks Architect — Spark, Delta Lake & Cloud

Vidha It Solutions Llc; Oregon; Hybrid; USD 120.000 - 160.000; Vollzeit

Zusammenfassung

Vidha IT Solutions LLC is seeking a Databricks Architect to design, build, and optimize scalable data pipelines using Databricks and Spark. The role involves Delta Lake governance, lakehouse patterns, and cross-functional collaboration with data scientists and analysts. Remote work is offered, with visa W2 eligibility.

The candidate will manage clusters, notebooks, and job scheduling while ensuring data quality and cost efficiency across AWS, Azure, or GCP environments.

Qualifikationen

• Proven experience as a Databricks Engineer or Data Engineer with hands-on Databricks platform expertise.
• Strong proficiency in Apache Spark, PySpark, and SQL.
• Experience with Delta Lake, Lakehouse architecture, and data modeling.
• Familiarity with cloud platforms (AWS, Azure, or GCP) and cloud-native data services.
• Experience building and managing ETL/ELT pipelines at scale.
• Knowledge of Python, Scala, or other relevant programming languages.
• Strong problem-solving skills and ability to work in a collaborative, fast-paced environment.

Aufgaben

• Design, develop, and maintain scalable data pipelines using Databricks and Apache Spark.
• Build and optimize ETL/ELT workflows to support data ingestion, transformation, and delivery.
• Collaborate with data scientists, analysts, and business stakeholders to deliver data-driven solutions.
• Manage and monitor Databricks clusters, notebooks, and job scheduling for performance and cost efficiency.
• Implement Delta Lake architecture and best practices for data reliability and governance.
• Integrate Databricks with cloud platforms such as AWS, Azure, or GCP and enterprise data sources.
• Troubleshoot and resolve data pipeline issues, ensuring high availability and data quality.

Kenntnisse

Databricks

Apache Spark

PySpark

SQL

Python

Scala

Delta Lake

Lakehouse

Data modeling

Cloud data services

Tools

Delta Lake

AWS

Azure

GCP

Jobbeschreibung

Vidha IT Solutions LLC is seeking a Databricks Architect to design, build, and optimize scalable data pipelines using Databricks and Spark. The role involves Delta Lake governance, lakehouse patterns, and cross-functional collaboration with data scientists and analysts. Remote work is offered, with visa W2 eligibility.

The candidate will manage clusters, notebooks, and job scheduling while ensuring data quality and cost efficiency across AWS, Azure, or GCP environments.