Senior Data Engineer - Lakehouse & Streaming Pipelines

MACHINE LEARNING TECHNOLOGIES LLC; Atlanta (GA); Vor Ort; USD 140.000 - 190.000; Vollzeit

Zusammenfassung

MACHINE LEARNING TECHNOLOGIES LLC in Atlanta, GA is seeking a Senior Data Engineer to design, build and operate production-grade batch and streaming data pipelines across SQL Server, cloud applications, device/event streams, and CRM sources.

You will implement lakehouse layers with Iceberg and Trino, develop data models for buildings, tenants and devices, ensure data quality, monitoring, and security, and own CI/CD, IaC, and on-call activities with global teams.

Qualifikationen

• 6+ years in data engineering or data platform development.
• Expert SQL with performance tuning and complex queries.
• Proficiency in Python, Java, or Scala.
• Hands-on with Iceberg, Delta Lake, or Hudi.
• Experience with Trino/Presto/Spark SQL and Snowflake.
• Work with streaming and CDC tools like Flink and Kafka.
• Strong dimensional modeling, data quality, and data contracts.
• Kubernetes, CI/CD, and infrastructure-as-code experience.
• Ability to troubleshoot complex production data issues.

Aufgaben

• Build and operate production-grade batch and streaming data pipelines.
• Develop ingestion, transformation, and enrichment workflows.
• Implement lakehouse layers with Iceberg and Trino across zones.
• Model dimensional structures for buildings, tenants, devices, and people.
• Ensure data quality checks, monitoring, lineage, and alerts.
• Maintain integration between lakehouse and Snowflake with secure sharing.
• Own CI/CD pipelines, automated testing, and IaC for deployments.
• Provide production support, incident response, and RCA.
• Collaborate with Product, QA, DevOps, Analytics, and distributed teams.

Kenntnisse

Strong SQL

Python/Java/Scala

Apache Flink

Kafka

Snowflake

Trino / Presto / Spark SQL

Apache Iceberg

Kubernetes / Containers

Data Modeling

Data Quality & Monitoring

Tools

Apache Iceberg

Delta Lake

Apache Hudi

Trino

Presto

Spark SQL

Snowflake

Kubernetes

Infrastructure as Code

Jobbeschreibung

MACHINE LEARNING TECHNOLOGIES LLC in Atlanta, GA is seeking a Senior Data Engineer to design, build and operate production-grade batch and streaming data pipelines across SQL Server, cloud applications, device/event streams, and CRM sources.

You will implement lakehouse layers with Iceberg and Trino, develop data models for buildings, tenants and devices, ensure data quality, monitoring, and security, and own CI/CD, IaC, and on-call activities with global teams.