Senior Data Engineer

Highbrow LLC; United States; Remote; USD 140.000 - 190.000; Vollzeit

Zusammenfassung

Highbrow LLC seeks a Senior Data Engineer for a remote role supporting Kastle System. You will build and operate batch and streaming pipelines across SQL Server, cloud apps, device/event streams, and CRM systems, with strong Snowflake experience.

You will develop ingestion, transformation, and enrichment pipelines using Flink, Kafka, CDC, SQL, and Python/Java/Scala, and implement lakehouse layers with Iceberg and Trino.

Qualifikationen

• 6+ years of hands-on experience in data engineering or data platform development.
• Strong SQL and data warehousing capabilities.
• Experience with Snowflake and lakehouse architectures.

Aufgaben

• Build and operate production-grade batch and streaming data pipelines across SQL Server, cloud applications, device/event streams, CRM systems, and third-party sources.
• Develop ingestion, transformation, and enrichment pipelines using Flink, Kafka, CDC, SQL, and Python/Java/Scala.
• Implement lakehouse data layers using Iceberg and Trino.
• Build dimensional and analytical models for buildings, tenants, credentials, devices, and people.
• Implement CDC, late-arriving data, retries, backfills, schema evolution, and data reconciliation.
• Integrate lakehouse with Snowflake, including Iceberg-backed tables, secure sharing, and data contracts.
• Develop automated data-quality checks, pipeline monitoring, lineage, and dashboards.
• Troubleshoot production data issues and optimize queries, storage, and compute.
• Implement security controls and CI/CD pipelines, IaC, and environment promotions.
• Participate in production support, incident resolution, and on-call duties.
• Collaborate with Product, QA, DevOps, Analytics, and distributed teams.

Kenntnisse

SQL

Python/Java/Scala

Data modeling

Performance tuning

Data quality

CI/CD

Cloud data platforms

Ausbildung

Bachelor’s or Master’s Degree in CS/EE/Math

Tools

Apache Iceberg

Delta Lake

Apache Hudi

Trino

Spark SQL

Snowflake

Kafka

Flink

Debezium

Kubernetes

dbt

Jobbeschreibung

Senior Data Engineer (Client: Kastle System) [Remote. Strong Snowflake experience is a must-have; other details are provided below.]

Location: Remote

Note: Strong Snowflake experience is a must-have; other details are provided below.

Key Responsibilities:

• Build and operate production-grade batch and streaming data pipelines across SQL Server, cloud applications, device/event streams, Customer Relationship Management systems, and third-party sources.
• Develop ingestion, transformation, and enrichment pipelines using Apache Flink, Kafka, Change Data Capture, SQL, and Python/Java/Scala.
• Implement lakehouse data layers across landing, raw, conformed, and consumption zones using Apache Iceberg and Trino.
• Build dimensional and analytical models for entities such as buildings, tenants, credentials, devices, and people.
• Implement robust handling for Change Data Capture, late-arriving data, retries, backfills, replay, schema evolution, and data reconciliation.
• Build and maintain integration between the lakehouse and Snowflake, including Iceberg-backed/external tables, secure sharing, and downstream data contracts.
• Develop automated data-quality checks, pipeline monitoring, lineage, freshness alerts, and operational dashboards.
• Troubleshoot and resolve production data issues, pipeline failures, performance bottlenecks, and data-quality problems.
• Optimize SQL queries, storage layouts, partitioning, clustering, compute utilization, and overall platform cost.
• Implement security controls including role-based access, row/column-level security, masking, and protection of sensitive data.
• Build and maintain Continuous Integration and Continuous Deployment pipelines, automated testing, infrastructure-as-code, and environment promotion processes.
• Participate in production support, incident resolution, Root Cause Analysis, deployment, and on-call activities.
• Work closely with Product, Application Engineering, Quality Assurance, DevOps/Site Reliability Engineering, Analytics, and globally distributed engineering teams.
• Contribute to technical design decisions and documentation while remaining primarily hands-on in implementation and production delivery.

Required Skills:

• 6+ years of hands-on experience in data engineering, data warehousing, or data platform development.
• Strong production experience building and supporting end-to-end data pipelines.
• Expert SQL, including complex queries, query-plan analysis, and performance tuning.
• Strong development skills in Python, Java, or Scala.
• Hands-on experience with Apache Iceberg, Delta Lake, or Apache Hudi.
• Experience with distributed query technologies such as Trino, Presto, or Spark SQL.
• Hands-on Snowflake experience, including data modeling, performance optimization, external/Iceberg tables, secure data sharing, or access management.
• Experience with streaming and Change Data Capture technologies such as Apache Flink, Kafka, Spark Structured Streaming, or Debezium.
• Strong understanding of dimensional modeling, data quality, schema evolution, and data contracts.
• Experience with Kubernetes, containers, Continuous Integration and Continuous Deployment, and infrastructure-as-code.
• Experience implementing monitoring, alerting, lineage, reconciliation, and operational support for production data platforms.
• Ability to independently troubleshoot complex production issues and drive them through resolution.

Preferred Qualifications:

• Experience with multi-tenant Software as a Service data platforms and customer data isolation.
• Experience integrating open lakehouse platforms with Snowflake.
• Azure cloud experience.
• Experience with Airflow, Dagster, dbt, Power BI, Looker, or Superset.
• Experience with data governance, cataloging, masking, and access-control frameworks.
• Exposure to Internet of Things/device telemetry, physical security, commercial real estate, or property technology.
• Experience using Artificial Intelligence-assisted engineering tools to accelerate data development.

Education:

• Bachelor’s or Master’s Degree in Computer Science, Computer or Electrical Engineering, Mathematics, or a related field.