Data Engineer

Ilum; United States; Remote; USD 70,000 - 110,000; Full time

Job summary

An innovative company is seeking a skilled Data Engineer to enhance its data lakehouse platform. This role involves collaborating with cross-functional teams to integrate advanced data storage solutions and optimize performance for real-time insights. You'll play a vital role in supporting AI initiatives while ensuring data quality and compliance. Join a globally distributed team dedicated to transforming data management and driving AI innovation. If you're passionate about data engineering and want to make a significant impact, this opportunity is perfect for you.

Benefits

Competitive Compensation

Fully Remote Work Environment

Professional Growth Opportunities

Training and Certification Programs

Qualifications

• Proven experience in building and maintaining data pipelines.
• Proficiency in Apache Spark and familiarity with big data technologies.

Responsibilities

• Collaborate with teams to integrate data storage solutions into the platform.
• Implement data governance practices to ensure quality and compliance.

Skills

Data Pipeline Development

Apache Spark

ETL Processes

SQL

Data Modeling

Communication Skills

Tools

Delta Lake

Apache Iceberg

Apache Hudi

PowerBI

Tableau

AWS

Azure

GCP

Apache Airflow

MLflow

Job description

Get AI-powered advice on this job and more exclusive features.

Location: Full Remote – Work from Anywhere Globally

Ilum is a pioneering free data lakehouse designed to empower organizations through data and AI transformation. Built on Apache Spark (with Trino and Flink coming soon), Ilum offers a highly customizable platform featuring integrated data cataloging, automatic data lineage tracking, and live SQL support. Our mission is to simplify data management and enable rapid AI innovation. Join us and work with a dynamic, globally distributed team that’s redefining modern data infrastructure.

Recruitment Process (Please read this):

Due to the high volume of applicants, only candidates who complete the following task will move to the next step. The first stage of our recruitment process involves a small project designed to showcase your skills as a software/data engineer. This task is broad, it’s meant to evaluate your innovation, creativity, and technical prowess using Ilum. You can approach it in any form that best demonstrates your talents: a use case, blog post, training film, proposal for improvements, a business problem solution, or code that addresses a specific challenge. We encourage you to surprise us with your approach!

Important: Please host your project/code on GitHub. For inspiration, check out our top example from one of our current employees: GitHub Example.

Job Description:

We are seeking a skilled Data Engineer to join our growing team at Ilum. You will play a critical role in developing and optimizing our data lakehouse platform. In this role, you'll work closely with data architects, ML engineers, and product developers to build scalable, efficient, and reliable data systems that support our cutting-edge features and AI initiatives.

Key Responsibilities:

• Platform Integration: Collaborate with cross-functional teams to integrate data storage solutions (e.g., Delta Lake, Apache Iceberg, Apache Hudi) and BI tools (e.g., PowerBI, Tableau) into the Ilum ecosystem.
• Data Governance: Implement automatic data lineage and integrated data cataloging to ensure data quality, compliance, and traceability.
• Performance Optimization: Fine-tune system performance for live SQL queries and data processing workloads to ensure real-time insights.
• Innovation in Data & AI: Support our AI initiatives by designing and maintaining infrastructure that accelerates machine learning pipelines and advanced analytics.
• Global Collaboration: Work with team members from around the world to share best practices, participate in code reviews, and continuously enhance our data engineering processes.

Required Qualifications:

• Proven experience in building and maintaining data pipelines.
• Proficiency in Apache Spark and big data technologies, familiarity with Trino, Flink, or similar frameworks is a plus.
• Solid knowledge of ETL processes, SQL, and data modeling.
• Familiarity with data cataloging, data lineage tools, and live query optimization.

Preferred Qualifications:

• Experience with cloud platforms (AWS, Azure, or GCP) and hybrid data environments.
• Familiarity with orchestration tools like Apache Airflow and ML lifecycle tools like MLflow.
• Excellent communication skills and a passion for data and AI transformation.
• Experience with modern data storage solutions and BI integration.

What We Offer:

• Competitive compensation and a fully remote, globally distributed work environment.
• A collaborative, innovative team and ample opportunities for professional growth.
• Training and certification trainings.

If you're passionate about data engineering and excited to be at the forefront of data and AI transformation, we’d love to see your work and hear from you!

Seniority level

• Entry level

Employment type

• Full-time

Job function

• Information Technology
• Software Development