Senior Databricks Data Architect — Lakehouse & Pipelines

Incedo Inc.; San Rafael (CA); On-site; USD 100,000 - 140,000; Full time

Job summary

A leading data solutions company is seeking a Senior Talent Acquisition Specialist to design and optimize cloud-native data platforms. You will play a crucial role in collaborating with client stakeholders to create high-performance data architectures. The ideal candidate should have deep expertise in Apache Spark, Delta Lake, and experience with Databricks. This is a hands-on role focused on engineering solutions that adhere to best practices in data quality and governance, ensuring efficient data processing in cloud environments.

Qualifications

• Deep experience with Apache Spark and Delta Lake.
• Experience in building and managing Delta Lake tables.
• Hands-on experience in developing scalable data pipelines using Databricks.

Responsibilities

• Design, develop, and maintain batch and near-real-time data pipelines.
• Implement Medallion Lakehouse architectures ensuring data quality.
• Troubleshoot and resolve Spark job performance bottlenecks.

Skills

Apache Spark

Databricks

Delta Lake

PySpark

Spark SQL

Tools

Azure Data Factory

Apache Airflow

Power BI

Tableau

Looker

Job description

A leading data solutions company is seeking a Senior Talent Acquisition Specialist to design and optimize cloud-native data platforms. You will play a crucial role in collaborating with client stakeholders to create high-performance data architectures. The ideal candidate should have deep expertise in Apache Spark, Delta Lake, and experience with Databricks. This is a hands-on role focused on engineering solutions that adhere to best practices in data quality and governance, ensuring efficient data processing in cloud environments.