Apache Iceberg Engineer

Smart IT Frame LLC

Sunnyvale (CA)

On-site

USD 90,000 - 150,000

Full time

14 days+

Get a reply from this employer

Your resume and cover letter, tailored to exactly what they’re hiring for.

Get past ATS filters

Job summary

An innovative firm is seeking an experienced Apache Iceberg Engineer to design and optimize large-scale data lakehouse solutions. This role involves collaborating with data engineers and scientists to create efficient data architectures using cutting-edge technologies like Apache Spark and cloud storage solutions. The ideal candidate will possess strong expertise in big data frameworks, data governance, and cloud platforms. Join a forward-thinking team that values creativity and technical excellence, and make a significant impact on data management practices in a dynamic environment. If you're passionate about data engineering and eager to tackle complex challenges, this opportunity is perfect for you.

Qualifications

  • 3+ years of experience in Big Data, Data Engineering, or Cloud Data Warehousing.
  • Hands-on experience with Apache Iceberg in a production environment.

Responsibilities

  • Design and optimize Iceberg-based data lake architectures for large-scale datasets.
  • Develop data ingestion and transformation pipelines using Spark and Flink.

Skills

Apache Iceberg
Big Data Processing
Apache Spark
Flink
SQL
Data Governance
Cloud Storage Solutions
Data Lakehouse Architectures
Python
Java

Education

Bachelor's degree in Computer Science
Master's degree in Data Engineering

Tools

AWS S3
Google Cloud Storage
Azure Data Lake Storage
Terraform
Kubernetes
Airflow

Job description

We are looking for an experienced Apache Iceberg Engineer to design, develop, and optimize large-scale data lakehouse solutions leveraging Apache Iceberg. The ideal candidate will have expertise in big data processing frameworks (Apache Spark, Flink, Presto, Trino, Hive) and cloud-based data platforms like AWS S3, Google Cloud Storage, or Azure Data Lake Storage. You will work closely with data engineers, data scientists, and DevOps teams to ensure efficient, scalable, and reliable data architecture.

Key Responsibilities:

  • Design, implement, and optimize Iceberg-based data lake architectures for large-scale datasets.
  • Develop data ingestion, transformation, and query optimization pipelines using Spark, Flink, or Presto/Trino.
  • Ensure ACID compliance, schema evolution, and partition evolution in Iceberg tables.
  • Implement time travel, versioning, and snapshot management for historical data analysis.
  • Optimize metadata management and query performance in Iceberg-based data lakes.
  • Integrate Apache Iceberg with cloud storage solutions (AWS S3, GCS, ADLS) and data warehouses.
  • Implement best practices for data governance, access control, and security within an Iceberg-based environment.
  • Troubleshoot performance issues, metadata inefficiencies, and schema inconsistencies in Iceberg tables.
  • Collaborate with DevOps, ML engineers, and BI teams to enable smooth data workflows.

Required Qualifications:

  • Bachelor's or Master's degree in Computer Science, Data Engineering, or a related field.
  • 3+ years of experience in Big Data, Data Engineering, or Cloud Data Warehousing.
  • Hands-on experience with Apache Iceberg in a production environment.
  • Strong expertise in Apache Spark, Flink, Trino, Presto, or Hive for big data processing.
  • Proficiency in SQL and distributed query engines.
  • Experience working with cloud storage solutions (AWS S3, GCS, ADLS).
  • Knowledge of data lakehouse architectures and modern data management principles.
  • Familiarity with schema evolution, ACID transactions, and partitioning techniques.
  • Experience with Python, Scala, or Java for data processing.

Preferred Qualifications:

  • Experience in real-time data processing using Flink or Kafka.
  • Understanding of data governance, access control, and compliance frameworks.
  • Knowledge of other data lake frameworks like Delta Lake (Databricks) or Apache Hudi.
  • Hands-on experience with Terraform, Kubernetes, or Airflow for data pipeline automation.
Seniority level

Mid-Senior level

Employment type

Full-time

Job function

Other

Industries

Software Development and IT Services and IT Consulting

Get your free, confidential resume review.

or drag and drop your file here.

Similar jobs

Similar jobs worth comparing

Data Architect
Data Architect

EXL Talent Acquisition Team • Pittsburgh

On-site
USD 140,000 - 180,000
Data Architect
Data Architect

EXL • Dallas (TX)

Hybrid
USD 130,000 - 190,000
Data Architect
Data Architect

EXL • Pittsburgh

On-site
USD 140,000 - 190,000
Data Architect
Data Architect

Wise Skulls • Dallas (TX)

Hybrid
USD 140,000 - 190,000
Hybrid work model
Data Architect + Apache Iceberg
Data Architect + Apache Iceberg

EXL • Dallas (TX)

Hybrid
USD 130,000 - 150,000
Health insurance
Senior Data Architect: Lakehouse & Iceberg Lead (Hybrid)
Senior Data Architect: Lakehouse & Iceberg Lead (Hybrid)

EXL • Dallas (TX)

Hybrid
USD 130,000 - 190,000
Senior Lead Software Engineer
Senior Lead Software Engineer

Virtusa Corporation • New York (NY)

On-site
USD 140,000 - 190,000
Senior Data Engineer
Senior Data Engineer

CIS Technologies INC • United States

Remote
USD 120,000 - 160,000
Lakehouse Data Architect: Iceberg & Object Storage Lead
Lakehouse Data Architect: Iceberg & Object Storage Lead

EXL • Pittsburgh

On-site
USD 140,000 - 190,000
Data Engineer
Data Engineer

Compunnel Inc. • Dallas (TX)

On-site
USD 90,000 - 120,000