Senior Data Platform Engineer - Lakehouse & Streaming

Quantum Technologies. LLC; Atlanta (GA); On-site; USD 120,000 - 180,000; Full time

Job summary

Quantum Technologies LLC in Atlanta, GA is seeking an experienced Data Engineer to design and operate production-grade batch and streaming data pipelines across SQL Server, cloud apps, device streams, and CRM sources.

You will implement lakehouse layers with Apache Iceberg and Trino, build dimensional models for key entities, and ensure data quality, lineage, and secure data sharing with Snowflake. Collaborate with global teams and drive CI/CD and IaC initiatives.

Qualifications

• 6+ years of hands-on experience in data engineering, data warehousing, or data platform development.
• Strong production experience building and supporting end-to-end data pipelines.
• Expert SQL, including complex queries, query-plan analysis, and performance tuning.
• Strong development skills in Python, Java, or Scala.
• Hands-on experience with Apache Iceberg, Delta Lake, or Apache Hudi.
• Experience with distributed query technologies such as Trino, Presto, or Spark SQL.
• Hands-on Snowflake experience, including data modeling, performance optimization, external/Iceberg tables, secure data sharing, or access management.
• Experience with streaming and Change Data Capture technologies such as Apache Flink, Kafka, Spark Structured Streaming, or Debezium.
• Strong understanding of dimensional modeling, data quality, schema evolution, and data contracts.
• Experience with Kubernetes, containers, CI/CD, and infrastructure-as-code.

Responsibilities

• Build and operate production-scale batch and streaming data pipelines across multiple sources.
• Develop ingestion, transformation, and enrichment pipelines using Flink, Kafka, SQL, and Python/Java/Scala.
• Implement lakehouse data layers using Iceberg and Trino across landing to consumption zones.
• Create dimensional and analytical models for buildings, tenants, credentials, devices, and people.
• Ensure data quality, reconciliation, schema evolution, and robust CDC handling.
• Integrate lakehouse with Snowflake through Iceberg-backed/external tables and secure sharing.
• Develop monitoring, lineage, freshness alerts, and operational dashboards.
• Troubleshoot production data issues, performance bottlenecks, and data quality problems.
• Optimize queries, storage layouts, partitions, and compute cost.
• Implement security controls including role-based access and data masking.
• Contribute to CI/CD pipelines, IaC, testing, and environment promotion.
• Participate in on-call, incident resolution, and root cause analysis.
• Collaborate with product, QA, DevOps, analytics, and distributed teams.

Skills

Data Engineering

SQL Proficiency

Python/Java/Scala

Apache Iceberg

Trino/Presto

Snowflake

Kafka

Flink

Kubernetes

Dimensional Modeling

Tools

Iceberg

Snowflake

Kafka

Airflow

Job description

Quantum Technologies LLC in Atlanta, GA is seeking an experienced Data Engineer to design and operate production-grade batch and streaming data pipelines across SQL Server, cloud apps, device streams, and CRM sources.

You will implement lakehouse layers with Apache Iceberg and Trino, build dimensional models for key entities, and ensure data quality, lineage, and secure data sharing with Snowflake. Collaborate with global teams and drive CI/CD and IaC initiatives.