IoT Data Platform Architect – Real-Time Pipelines

CRG Search; Fort Mill (SC); Ibrido; USD 170.000 - 210.000; Tempo pieno

Descrizione del lavoro

CRG Search seeks a Senior Data Engineer – IoT to lead the design and implementation of a scalable IoT data platform. You will build end‑to‑end data pipelines from edge devices into Databricks, enabling real‑time analytics, AI/ML, and customer‑facing insights.

The role blends hands‑on engineering with architectural direction, mentoring junior engineers and shaping long‑term IoT data strategy across the organization.

Competenze

• 5+ years of hands‑on IoT or large‑scale device data experience.
• Senior data engineering experience with high/real‑time data.
• Proficiency in Python and SQL; production Spark/PySpark work.

Mansioni

• Design, build, and maintain end‑to‑end IoT data pipelines into Databricks.
• Lead IoT data migration from third‑party apps to Databricks.
• Develop streaming ingestion using Kafka, Event Hubs, and Structured Streaming.
• Establish data quality, schema governance, and scalable dev standards.
• Architect and optimize Delta Lake tables, including partitioning and cost controls.
• Collaborate with infra and cloud teams on Databricks clusters and Unity Catalog.
• Handle multi‑terabyte telemetry and time‑series data.
• Partner with data scientists and analysts to translate requirements.
• Ensure IoT data is ready for AI/ML, analytics, and downstream use cases.
• Mentor junior engineers with code reviews and guidance.
• Troubleshoot issues and implement long‑term solutions.
• Evaluate new technologies and IoT data strategy.

Conoscenze

Python

SQL

Spark/PySpark

Databricks

Streaming

Kafka

Azure Event Hubs

Delta Lake

Unity Catalog

Delta Live Tables

Databricks Workflows

Data modeling

Formazione

Bachelor's degree in CS/IT/Data Engineering

Equivalent professional experience

Strumenti

Databricks

Apache Kafka

Azure Event Hubs

Delta Lake

Unity Catalog

Delta Live Tables

Databricks Workflows

Descrizione del lavoro

CRG Search seeks a Senior Data Engineer – IoT to lead the design and implementation of a scalable IoT data platform. You will build end‑to‑end data pipelines from edge devices into Databricks, enabling real‑time analytics, AI/ML, and customer‑facing insights.

The role blends hands‑on engineering with architectural direction, mentoring junior engineers and shaping long‑term IoT data strategy across the organization.