Senior Data Platform Engineer: Lakehouse, Streaming & CI/CD

Robotic Technologies Automation; Atlanta (GA); On-site; USD 120,000 - 180,000; Full time

Job summary

Robotic Technologies Automation seeks an experienced data engineer to design and operate production-grade data pipelines across batch and streaming sources. You will implement lakehouse layers using Iceberg and Trino, model dimensional structures, and ensure data quality and security.

Role includes building CI/CD, IaC, and collaborating with global teams to deliver reliable data platforms and dashboards. Strong SQL and Spark ecosystem experience required.

Qualifications

• 6+ years of hands-on experience in data engineering, data warehousing, or data platform development.
• Strong production experience building and supporting end-to-end data pipelines.
• Expert SQL, including complex queries, query-plan analysis, and performance tuning.
• Strong development skills in Python, Java, or Scala.
• Hands-on experience with Apache Iceberg, Delta Lake, or Apache Hudi.
• Experience with distributed query technologies such as Trino, Presto, or Spark SQL.
• Hands-on Snowflake experience, including data modeling, performance optimization, external/Iceberg tables, secure data sharing, or access management.
• Experience with streaming and Change Data Capture technologies such as Apache Flink, Kafka, Spark Structured Streaming, or Debezium.
• Strong understanding of dimensional modeling, data quality, schema evolution, and data contracts.
• Experience with Kubernetes, containers, Continuous Integration and Continuous Deployment, and infrastructure-as-code.

Responsibilities

• Build and operate production-grade batch and streaming data pipelines across SQL Server, cloud applications, device/event streams, CRMs, and third-party sources.
• Develop ingestion, transformation, and enrichment pipelines using Flink, Kafka, CDC, SQL, and Python/Java/Scala.
• Implement lakehouse data layers across landing, raw, conformed, and consumption zones using Iceberg and Trino.
• Build dimensional and analytical models for entities such as buildings, tenants, credentials, devices, and people.
• Implement robust handling for CDC, late-arriving data, retries, backfills, replay, schema evolution, and data reconciliation.
• Integrate lakehouse with Snowflake, including Iceberg-backed/external tables and secure data sharing.
• Develop automated data-quality checks, pipeline monitoring, lineage, freshness alerts, and dashboards.
• Troubleshoot production data issues, pipeline failures, performance bottlenecks, and data-quality problems.
• Optimize SQL queries, storage layouts, partitioning, clustering, compute utilization, and platform cost.
• Implement security controls including RBAC, row/column-level security, masking, and data protection.
• Build and maintain CI/CD pipelines, automated testing, IaC, and environment promotion processes.
• Participate in production support, incident resolution, RCA, deployment, and on-call activities.
• Collaborate with Product, Application Engineering, QA, DevOps/SRE, Analytics, and distributed teams.
• Contribute to design decisions and documentation while remaining hands-on in implementation and delivery.

Skills

SQL performance tuning

End-to-end data pipelines

Production issue troubleshooting

Dimensional modeling

Data quality

Python/Java/Scala

Data warehousing

Tools

Apache Iceberg

Delta Lake

Apache Hudi

Trino/Presto

Spark SQL

Kafka

Flink

Snowflake

Airflow

Dagster

dbt

Power BI

Looker

Superset

Kubernetes

CI/CD

Infrastructure as Code

Job description

Robotic Technologies Automation seeks an experienced data engineer to design and operate production-grade data pipelines across batch and streaming sources. You will implement lakehouse layers using Iceberg and Trino, model dimensional structures, and ensure data quality and security.

Role includes building CI/CD, IaC, and collaborating with global teams to deliver reliable data platforms and dashboards. Strong SQL and Spark ecosystem experience required.