On-Prem Lakehouse Architect - Data Platform Lead

Innovaccer Inc.; United States; On-site; USD 180,000 - 260,000; Full time

Benefits offered by this job

20 days PTO

Parental leave

Rewards program

Insurance coverage

Job summary

Innovaccer Inc. in the United States is seeking a Principal Software Engineer to own the lakehouse engine architecture for on-prem deployments using Iceberg, Trino, a REST catalog, and Spark.

You will design, guide, and optimize the integration of catalog, engine placement, and pipelines, while mentoring senior engineers and collaborating with platform teams to ensure scalable performance and reliability.

Qualifications

• 12+ years experience building and operating large-scale data platforms or distributed systems.
• Deep hands-on expertise with distributed SQL engines (Trino/Presto or Spark SQL).
• Experience with Apache Iceberg and table maintenance at scale.
• Knowledge of Iceberg catalogs (REST catalogs or Hive Metastore).
• Understanding of cloud data warehouse internals (Snowflake/BigQuery/Redshift) to design equivalents.
• Proficiency in Java and/or Python.

Responsibilities

• Own the lakehouse reference architecture (Iceberg table design, Trino cluster topology, catalog service, Spark transform compute, object-storage layout).
• Design on-premise replacements for cloud-managed warehouse capabilities.
• Run PoC validation and define triggers for placement decisions (VM vs Kubernetes).
• Set platform-wide standards for table layout, partitioning, and Iceberg maintenance.
• Lead SQL dialect strategy for porting workloads to Trino and Spark SQL.
• Mentor senior engineers and collaborate with platform teams on storage sizing and capacity planning.

Skills

Distributed SQL

Java

Python

Iceberg

Trino

Spark SQL

Iceberg Catalogs

Education

B.E./B.Tech./M.Sc. in CS

Tools

Polaris

Nessie

S3-compatible storage

Job description

Innovaccer Inc. in the United States is seeking a Principal Software Engineer to own the lakehouse engine architecture for on-prem deployments using Iceberg, Trino, a REST catalog, and Spark.

You will design, guide, and optimize the integration of catalog, engine placement, and pipelines, while mentoring senior engineers and collaborating with platform teams to ensure scalable performance and reliability.