Senior Data Platform Engineer Azure Lakehouse & Kubernetes

Cloud Analytics Technologies, LLC; Sunnyvale (CA); On-site; USD 130,000 - 210,000; Full time

Job summary

Cloud Analytics Technologies, LLC in Sunnyvale, CA is seeking a capable Data Platform Engineer to design, deploy, and operate data infrastructure on Kubernetes in Microsoft Azure. You will build lakehouse pipelines with Iceberg on ADLS, tune Trino and Spark, and implement data governance across multiple data stores.

You will develop DBT transformations with robust validation, automation via IaC/CI/CD, and collaborate with data engineers and analysts to optimize workloads and scale the platform.

Qualifications

• Hands-on experience deploying Kubernetes and stateful workloads.
• Experience with Microsoft Azure cloud and ADLS.
• Familiarity with Apache Iceberg and Iceberg catalogs REST.
• Expertise with Trino and Spark for distributed compute.
• Hands-on DBT for data transformation and analytics engineering.
• Solid understanding of data lake and lakehouse architectures and file formats.

Responsibilities

• Design, deploy, and operate data platform services on Kubernetes in Microsoft Azure.
• Build and manage lakehouse infrastructure on Iceberg backed by ADLS, incl. catalog config, schema evolution, and partitioning.
• Operate and tune Trino and Spark clusters for performance and cost efficiency.
• Develop and maintain data transformation pipelines using DBT with testing and docs.
• Define and enforce standards for data access, security, governance, and multi-tenant resource management across ADLS and query engines.
• Implement observability for the platform, including monitoring, alerts, and incident response.
• Partner with data engineers, analysts, and platform teams to onboard workloads and resolve bottlenecks.
• Automate infrastructure and deployments using IaC and CI/CD.

Skills

Kubernetes

Azure Cloud

ADLS

Iceberg

Trino

Spark

DBT

Data governance

Parquet

ORC

Query optimization

Tools

CI/CD

IaC

Job description

Cloud Analytics Technologies, LLC in Sunnyvale, CA is seeking a capable Data Platform Engineer to design, deploy, and operate data infrastructure on Kubernetes in Microsoft Azure. You will build lakehouse pipelines with Iceberg on ADLS, tune Trino and Spark, and implement data governance across multiple data stores.

You will develop DBT transformations with robust validation, automation via IaC/CI/CD, and collaborate with data engineers and analysts to optimize workloads and scale the platform.