Senior Data Engineer — Lakehouse & Streaming Platform

Valiant Technologies LLC; Atlanta (GA); On-site; USD 120,000 - 180,000; Full time

Job summary

Valiant Technologies LLC is seeking an experienced data engineer to design and operate scalable data pipelines and lakehouse architectures in a production setting. You will work across SQL Server, cloud apps, device streams, and third-party sources to deliver reliable ingestion, transformation, and enrichment processes.

The role emphasizes hands-on implementation with Flink, Kafka, CDC, SQL, Python/Java/Scala, and building analytical models for various entities.

Qualifications

• 6+ years of hands-on experience in data engineering, data warehousing, or data platform development.
• Strong production experience building and supporting end-to-end data pipelines.
• Expert SQL, including complex queries, query-plan analysis, and performance tuning.
• Strong development skills in Python, Java, or Scala.
• Hands-on experience with Apache Iceberg, Delta Lake, or Apache Hudi.
• Experience with distributed query technologies such as Trino, Presto, or Spark SQL.
• Hands-on Snowflake experience, including data modeling, performance optimization, external/Iceberg tables, secure data sharing, or access management.
• Experience with streaming and Change Data Capture technologies such as Apache Flink, Kafka, Spark Structured Streaming, or Debezium.
• Strong understanding of dimensional modeling, data quality, schema evolution, and data contracts.
• Experience with Kubernetes, containers, Continuous Integration and Continuous Deployment, and infrastructure-as-code.
• Experience implementing monitoring, alerting, lineage, reconciliation, and operational support for production data platforms.
• Ability to independently troubleshoot complex production issues and drive them through resolution.

Responsibilities

• Build and operate production-grade batch and streaming data pipelines across SQL Server, cloud applications, device/event streams, and third-party sources.
• Develop ingestion, transformation, and enrichment pipelines using Flink, Kafka, CDC, SQL, and Python/Java/Scala.
• Implement lakehouse data layers using Iceberg and Trino across landing, raw, conformed, and consumption zones.
• Build dimensional and analytical models for entities like buildings, tenants, credentials, devices, and people.
• Implement robust CDC handling, late-arriving data, retries, backfills, replay, and schema evolution.
• Maintain integration between lakehouse and Snowflake, including Iceberg-backed/external tables and secure sharing.
• Develop automated data-quality checks, pipeline monitoring, lineage, and dashboards.
• Troubleshoot production data issues, pipeline failures, and performance bottlenecks.
• Optimize SQL queries, storage layouts, partitioning, clustering, and compute utilization.
• Implement security controls including RBAC, row/column-level security, and masking.
• Build CI/CD pipelines, automated testing, IaC, and environment promotion processes.
• Participate in production support, incident resolution, RCA, deployment, and on-call activities.
• Collaborate with Product, App Engineering, QA, DevOps/SRE, Analytics, and globally distributed teams.
• Contribute to design decisions and documentation while remaining hands-on.

Skills

Data engineering

SQL performance tuning

Python/Java/Scala

Iceberg/Delta/Hudi

Trino/Presto/Spark SQL

Snowflake

Flink/Kafka/CDC

Dimensional modeling

Kubernetes/CI/CD/infra-as-code

Monitoring/alerting

Tools

Airflow

Dagster

dbt

Power BI

Looker

Superset

Job description

Valiant Technologies LLC is seeking an experienced data engineer to design and operate scalable data pipelines and lakehouse architectures in a production setting. You will work across SQL Server, cloud apps, device streams, and third-party sources to deliver reliable ingestion, transformation, and enrichment processes.

The role emphasizes hands-on implementation with Flink, Kafka, CDC, SQL, Python/Java/Scala, and building analytical models for various entities.