Data Architect

Incedo Inc.; San Rafael (CA); On-site; USD 100,000 - 140,000; Full time

Job summary

A leading data solutions company is seeking a Senior Talent Acquisition Specialist to design and optimize cloud-native data platforms. You will play a crucial role in collaborating with client stakeholders to create high-performance data architectures. The ideal candidate should have deep expertise in Apache Spark, Delta Lake, and experience with Databricks. This is a hands-on role focused on engineering solutions that adhere to best practices in data quality and governance, ensuring efficient data processing in cloud environments.

Qualifications

• Deep experience with Apache Spark and Delta Lake.
• Experience in building and managing Delta Lake tables.
• Hands-on experience in developing scalable data pipelines using Databricks.

Responsibilities

• Design, develop, and maintain batch and near-real-time data pipelines.
• Implement Medallion Lakehouse architectures ensuring data quality.
• Troubleshoot and resolve Spark job performance bottlenecks.

Skills

Apache Spark

Databricks

Delta Lake

PySpark

Spark SQL

Tools

Azure Data Factory

Apache Airflow

Power BI

Tableau

Looker

Job description

Senior Talent Acquisition Specialist at Incedo - Hiring for Life Sciences

Role Overview

We are seeking a Data bricks Data Architect to support the design, implementation, and optimization of cloud-native data platforms built on the Data bricks Lakehouse Architecture. This is a hands-on, engineering-driven role requiring deep experience with Apache Spark, Delta Lake, and scalable data pipeline development, combined with early-stage architectural responsibilities.

The role involves close onsite collaboration with client stakeholders, translating analytical and operational requirements into robust, high-performance data architectures, while adhering to best practices for data modeling, governance, reliability, and cost efficiency.

Key Responsibilities

• Design, develop, and maintain batch and near-real-time data pipelines using Databricks, PySpark, and Spark SQL
• Implement Medallion (Bronze/Silver/Gold) Lakehouse architectures, ensuring proper data quality, lineage, and transformation logic across layers
• Build and manage Delta Lake tables, including schema evolution, ACID transactions, time travel, and optimized data layouts
• Apply performance optimization techniques such as partitioning strategies, Z-Ordering, caching, broadcast joins, and Spark execution tuning
• Support dimensional and analytical data modeling for downstream consumption by BI tools and analytics applications
• Assist in defining data ingestion patterns (batch, incremental loads, CDC, and streaming where applicable)
• Troubleshoot and resolve pipeline failures, data quality issues, and Spark job performance bottlenecks.

Nice-to-Have Skills

• Exposure to Data bricks Unity Catalog, data governance, and access control models
• Experience with Data bricks Workflows, Apache Airflow, or Azure Data Factory for orchestration
• Familiarity with streaming frameworks (Spark Structured Streaming, Kafka) and/or CDC patterns
• Understanding of data quality frameworks, validation checks, and observability concepts
• Experience integrating Data bricks with BI tools such as Power BI, Tableau, or Looker
• Awareness of cost optimization strategies in cloud-based data platforms