Lead Databricks Lakehouse Architect

SkyLambda Inc.; Irvine (CA); On-site; USD 140,000 - 190,000; Full time

Job summary

SkyLambda Inc. in Irvine, CA designs cloud data platforms and scalable lakehouse architectures.

We are seeking a senior data engineer to lead the design and implementation of data architectures on the Databricks Lakehouse Platform, and to optimize pipelines with Spark, PySpark, SQL, and Delta Lake. You will collaborate with data engineers, analysts, data scientists and business teams to deliver robust data solutions, establish architecture standards, mentor engineers, and drive security,

Responsibilities

• Design and implement scalable data architectures using the Databricks Lakehouse Platform.
• Develop and optimize data pipelines using Apache Spark, PySpark, SQL, and Delta Lake.
• Architect and implement ETL/ELT workflows for batch and real-time data processing.
• Design and manage data lakehouse solutions, including data ingestion, transformation, storage, and consumption layers.
• Implement data governance, security, access controls, data quality, and metadata management using Unity Catalog and related tools.
• Optimize Spark jobs, cluster configurations, query performance, and compute costs.
• Integrate Databricks with cloud platforms such as AWS, Microsoft Azure, or Google Cloud Platform.
• Design and implement workflow orchestration and scheduling using Databricks Workflows and tools such as Apache Airflow.
• Establish CI/CD pipelines, infrastructure automation, and deployment best practices using tools such as Git, Terraform, and cloud-native DevOps services.
• Implement monitoring, logging, error handling, and performance optimization for production data pipelines.
• Collaborate with data engineers, analysts, data scientists, and business teams to define technical requirements and deliver data solutions.
• Provide technical leadership, conduct code reviews, establish architecture standards, and mentor engineering teams.
• Ensure compliance with enterprise data security, privacy, governance, and regulatory requirements.
• Evaluate emerging data technologies and recommend improvements to the existing data platform.

Job description

SkyLambda Inc. in Irvine, CA designs cloud data platforms and scalable lakehouse architectures.

We are seeking a senior data engineer to lead the design and implementation of data architectures on the Databricks Lakehouse Platform, and to optimize pipelines with Spark, PySpark, SQL, and Delta Lake. You will collaborate with data engineers, analysts, data scientists and business teams to deliver robust data solutions, establish architecture standards, mentor engineers, and drive security,