Senior Data Engineer

Robotic Process Automation LLC; Atlanta (GA); On-site; USD 120,000 - 180,000; Full time

Job summary

Robotic Technologies Automation seeks an experienced data engineer to design and operate production-grade data pipelines across batch and streaming sources. You will implement lakehouse layers using Iceberg and Trino, model dimensional structures, and ensure data quality and security.

Role includes building CI/CD, IaC, and collaborating with global teams to deliver reliable data platforms and dashboards. Strong SQL and Spark ecosystem experience required.

Qualifications

• 6+ years of hands-on experience in data engineering, data warehousing, or data platform development.
• Strong production experience building and supporting end-to-end data pipelines.
• Expert SQL, including complex queries, query-plan analysis, and performance tuning.
• Strong development skills in Python, Java, or Scala.
• Hands-on experience with Apache Iceberg, Delta Lake, or Apache Hudi.
• Experience with distributed query technologies such as Trino, Presto, or Spark SQL.
• Hands-on Snowflake experience, including data modeling, performance optimization, external/Iceberg tables, secure data sharing, or access management.
• Experience with streaming and Change Data Capture technologies such as Apache Flink, Kafka, Spark Structured Streaming, or Debezium.
• Strong understanding of dimensional modeling, data quality, schema evolution, and data contracts.
• Experience with Kubernetes, containers, Continuous Integration and Continuous Deployment, and infrastructure-as-code.

Responsibilities

• Build and operate production-grade batch and streaming data pipelines across SQL Server, cloud applications, device/event streams, CRMs, and third-party sources.
• Develop ingestion, transformation, and enrichment pipelines using Flink, Kafka, CDC, SQL, and Python/Java/Scala.
• Implement lakehouse data layers across landing, raw, conformed, and consumption zones using Iceberg and Trino.
• Build dimensional and analytical models for entities such as buildings, tenants, credentials, devices, and people.
• Implement robust handling for CDC, late-arriving data, retries, backfills, replay, schema evolution, and data reconciliation.
• Integrate lakehouse with Snowflake, including Iceberg-backed/external tables and secure data sharing.
• Develop automated data-quality checks, pipeline monitoring, lineage, freshness alerts, and dashboards.
• Troubleshoot production data issues, pipeline failures, performance bottlenecks, and data-quality problems.
• Optimize SQL queries, storage layouts, partitioning, clustering, compute utilization, and platform cost.
• Implement security controls including RBAC, row/column-level security, masking, and data protection.
• Build and maintain CI/CD pipelines, automated testing, IaC, and environment promotion processes.
• Participate in production support, incident resolution, RCA, deployment, and on-call activities.
• Collaborate with Product, Application Engineering, QA, DevOps/SRE, Analytics, and distributed teams.
• Contribute to design decisions and documentation while remaining hands-on in implementation and delivery.

Skills

SQL performance tuning

End-to-end data pipelines

Production issue troubleshooting

Dimensional modeling

Data quality

Python/Java/Scala

Data warehousing

Tools

Apache Iceberg

Delta Lake

Apache Hudi

Trino/Presto

Spark SQL

Kafka

Flink

Snowflake

Airflow

Dagster

dbt

Power BI

Looker

Superset

Kubernetes

CI/CD

Infrastructure as Code

Job description

• Build and operate production-grade batch and streaming data pipelines across SQL Server, cloud applications, device/event streams, Customer Relationship Management systems, and third-party sources.
• Develop ingestion, transformation, and enrichment pipelines using Apache Flink, Kafka, Change Data Capture, SQL, and Python/Java/Scala.
• Implement lakehouse data layers across landing, raw, conformed, and consumption zones using Apache Iceberg and Trino.
• Build dimensional and analytical models for entities such as buildings, tenants, credentials, devices, and people.
• Implement robust handling for Change Data Capture, late-arriving data, retries, backfills, replay, schema evolution, and data reconciliation.
• Build and maintain integration between the lakehouse and Snowflake, including Iceberg-backed/external tables, secure sharing, and downstream data contracts.
• Develop automated data-quality checks, pipeline monitoring, lineage, freshness alerts, and operational dashboards.
• Troubleshoot and resolve production data issues, pipeline failures, performance bottlenecks, and data-quality problems.
• Optimize SQL queries, storage layouts, partitioning, clustering, compute utilization, and overall platform cost.
• Implement security controls including role-based access, row/column-level security, masking, and protection of sensitive data.
• Build and maintain Continuous Integration and Continuous Deployment pipelines, automated testing, infrastructure-as-code, and environment promotion processes.
• Participate in production support, incident resolution, Root Cause Analysis, deployment, and on-call activities.
• Work closely with Product, Application Engineering, Quality Assurance, DevOps/Site Reliability Engineering, Analytics, and globally distributed engineering teams.
• Contribute to technical design decisions and documentation while remaining primarily hands-on in implementation and production delivery.

Required Skills:

• 6+ years of hands-on experience in data engineering, data warehousing, or data platform development.
• Strong production experience building and supporting end-to-end data pipelines.
• Expert SQL, including complex queries, query-plan analysis, and performance tuning.
• Strong development skills in Python, Java, or Scala.
• Hands-on experience with Apache Iceberg, Delta Lake, or Apache Hudi.
• Experience with distributed query technologies such as Trino, Presto, or Spark SQL.
• Hands-on Snowflake experience, including data modeling, performance optimization, external/Iceberg tables, secure data sharing, or access management.
• Experience with streaming and Change Data Capture technologies such as Apache Flink, Kafka, Spark Structured Streaming, or Debezium.
• Strong understanding of dimensional modeling, data quality, schema evolution, and data contracts.
• Experience with Kubernetes, containers, Continuous Integration and Continuous Deployment, and infrastructure-as-code.
• Experience implementing monitoring, alerting, lineage, reconciliation, and operational support for production data platforms.
• Ability to independently troubleshoot complex production issues and drive them through resolution.

Preferred Qualifications:

• Experience with multi-tenant Software as a Service data platforms and customer data isolation.
• Experience integrating open lakehouse platforms with Snowflake.
• Experience with Airflow, Dagster, dbt, Power BI, Looker, or Superset.
• Experience with data governance, cataloging, masking, and access-control frameworks.
• Exposure to Internet of Things/device telemetry, physical security, commercial real estate, or property technology.
• Experience using Artificial Intelligence-assisted engineering tools to accelerate data development.

We are an equal opportunity employer. All aspects of employment including the decision to hire, promote, discipline, or discharge, will be based on merit, competence, performance, and business needs. We do not discriminate on the basis of race, color, religion, marital status, age, national origin, ancestry, physical or mental disability, medical condition, pregnancy, genetic information, gender, sexual orientation, gender identity or expression, national origin, citizenship/ immigration status, veteran status, or any other status protected under federal, state, or local law.