Senior Data Platform Engineer: Azure Lakehouse, Trino, Spark

Machine Learning Technologies, LLC; Sunnyvale (CA); On-site; USD 190,000 - 250,000; Full time

Job summary

Machine Learning Technologies, LLC is seeking a Data Platform Engineer in Sunnyvale, CA to design, deploy, and operate data platform services on Kubernetes in Microsoft Azure. The role builds lakehouse infrastructure on Apache Iceberg backed by ADLS and tunes Trino and Spark clusters for performance and cost efficiency.

The ideal candidate will develop DBT pipelines, enforce governance across ADLS and query engines, and implement observability with monitoring and incident response.

Qualifications

• Hands-on experience deploying and operating Kubernetes stateful and distributed workloads.
• Experience with Azure Cloud and ADLS.
• Hands-on with Apache Iceberg and catalogs REST.
• Expertise with Trino and Spark for distributed compute.
• DBT for data transformation and analytics engineering.
• Solid understanding of lakehouse architectures and file formats (Parquet/ORC).

Responsibilities

• Design, deploy, and operate data platform services on Kubernetes in Microsoft Azure.
• Build and manage lakehouse infrastructure on Apache Iceberg backed by ADLS, including catalog configuration and partitioning strategy.
• Operate and tune Trino and Spark clusters for performance and cost efficiency.
• Develop and maintain data transformation pipelines using DBT with solid modeling, testing, and documentation.
• Define and enforce data access, security, governance, and multi-tenant resource management across ADLS and query engines.
• Implement observability for the platform with monitoring, alerting, capacity planning and incident response.
• Collaborate with data engineers, analysts, and platform teams to onboard workloads and resolve performance bottlenecks.
• Automate infrastructure and deployments using infrastructure-as-code and CI/CD.

Skills

Distributed data platform knowledge

Cross-team collaboration

Tools

Kubernetes

Microsoft Azure Cloud

ADLS

Apache Iceberg

Iceberg catalogs REST

Trino

Spark

DBT

Parquet

ORC

Job description

Machine Learning Technologies, LLC is seeking a Data Platform Engineer in Sunnyvale, CA to design, deploy, and operate data platform services on Kubernetes in Microsoft Azure. The role builds lakehouse infrastructure on Apache Iceberg backed by ADLS and tunes Trino and Spark clusters for performance and cost efficiency.

The ideal candidate will develop DBT pipelines, enforce governance across ADLS and query engines, and implement observability with monitoring and incident response.