Senior Data Engineer - IoT

CRG Search; Fort Mill (SC); Ibrido; USD 170.000 - 210.000; Tempo pieno

Descrizione del lavoro

CRG Search seeks a Senior Data Engineer – IoT to lead the design and implementation of a scalable IoT data platform. You will build end‑to‑end data pipelines from edge devices into Databricks, enabling real‑time analytics, AI/ML, and customer‑facing insights.

The role blends hands‑on engineering with architectural direction, mentoring junior engineers and shaping long‑term IoT data strategy across the organization.

Competenze

• 5+ years of hands‑on IoT or large‑scale device data experience.
• Senior data engineering experience with high/real‑time data.
• Proficiency in Python and SQL; production Spark/PySpark work.

Mansioni

• Design, build, and maintain end‑to‑end IoT data pipelines into Databricks.
• Lead IoT data migration from third‑party apps to Databricks.
• Develop streaming ingestion using Kafka, Event Hubs, and Structured Streaming.
• Establish data quality, schema governance, and scalable dev standards.
• Architect and optimize Delta Lake tables, including partitioning and cost controls.
• Collaborate with infra and cloud teams on Databricks clusters and Unity Catalog.
• Handle multi‑terabyte telemetry and time‑series data.
• Partner with data scientists and analysts to translate requirements.
• Ensure IoT data is ready for AI/ML, analytics, and downstream use cases.
• Mentor junior engineers with code reviews and guidance.
• Troubleshoot issues and implement long‑term solutions.
• Evaluate new technologies and IoT data strategy.

Conoscenze

Python

SQL

Spark/PySpark

Databricks

Streaming

Kafka

Azure Event Hubs

Delta Lake

Unity Catalog

Delta Live Tables

Databricks Workflows

Data modeling

Formazione

Bachelor's degree in CS/IT/Data Engineering

Equivalent professional experience

Strumenti

Databricks

Apache Kafka

Azure Event Hubs

Delta Lake

Unity Catalog

Delta Live Tables

Databricks Workflows

Descrizione del lavoro

Schedule: Monday–Friday | Primarily On-Site with flexibility for 1–2 days remote

Compensation: $185,000 + bonus

Position Overview

Our client is seeking a Senior Data Engineer – IoT to take a leading role in the development and long-term direction of its IoT data platform. This is a newly created position for a highly hands‑on engineer who brings significant experience working with large‑scale IoT, machine, or telemetric data.

This individual will help drive the continued migration of IoT data into Databricks, while designing and building a scalable data platform that supports real‑time and near‑real‑time reporting, analytics, data science, AI/ML, customer‑facing applications, and sustainability initiatives.

This is an opportunity to have meaningful ownership over the organization's IoT data strategy—not simply implement an existing specification. The ideal candidate will be comfortable moving between hands‑on engineering, architecture, technical guidance, and business‑facing discussions.

Key Responsibilities

• Design, build, and maintain end‑to‑end IoT data pipelines from edge devices into Databricks using Bronze/Silver/Gold medallion architecture.
• Lead the migration of IoT data from a third‑party application into a scalable Databricks environment.
• Develop and optimize streaming ingestion pipelines using technologies such as Apache Kafka, Azure Event Hubs, and Databricks Structured Streaming.
• Establish data engineering best practices, including pipeline design patterns, data quality frameworks, schema governance, and scalable development standards.
• Architect and optimize Delta Lake tables, including partitioning, Z‑ordering, and data lifecycle management for performance and cost efficiency.
• Collaborate with infrastructure and cloud teams on Databricks clusters, Unity Catalog, Delta Live Tables, and related cloud technologies.
• Develop solutions capable of supporting multi‑terabyte, high‑volume telemetry and time‑series data.
• Partner with data scientists, analysts, and business stakeholders to translate requirements into scalable data solutions.
• Ensure IoT data is structured and optimized for downstream analytics, AI/ML, reporting, and customer‑facing use cases.
• Mentor junior and mid‑level data engineers through code reviews, pair programming, technical guidance, and architectural direction.
• Troubleshoot production pipeline issues, perform root‑cause analysis, and implement long‑term solutions.
• Evaluate emerging technologies and recommend improvements to the organization's IoT and data engineering platform.
• Represent the data engineering team in technical and business discussions and help shape the organization's long‑term IoT data strategy.

Required Qualifications

• 5+ years of hands‑on experience working with IoT, machine, telemetric, or similarly large‑scale device data.
• Strong senior‑level data engineering experience, preferably in environments involving high‑volume or real‑time data.
• Strong proficiency in Python and SQL, with significant experience developing production‑grade Spark/PySpark solutions.
• Experience designing and implementing data pipelines for high‑frequency telemetry, time‑series data, or device‑generated data.
• Experience with Databricks and/or other large‑scale cloud data platforms.
• Strong understanding of data architecture, pipeline design, data quality, and schema governance.
• Experience working with streaming or near‑real‑time data architectures.
• Ability to lead technical discussions and translate business requirements into scalable technical solutions.
• Strong communication and interpersonal skills, with the ability to work effectively with engineers, data scientists, analysts, and business stakeholders.
• Ability to provide technical guidance and mentorship to other engineers.

Preferred Qualifications

• Hands‑on experience with Databricks, PySpark, Delta Lake, Delta Live Tables, Unity Catalog, and Databricks Workflows.
• Experience with Apache Kafka, Azure Event Hubs, or similar streaming technologies.
• Experience with fleet telematics platforms such as Geotab or similar IoT/connected‑vehicle platforms.
• Experience working with Azure cloud infrastructure and infrastructure‑as‑code concepts.
• Familiarity with Azure Key Vault and Azure Data Factory.
• Experience with dbt or other modern data transformation tools.
• Databricks Certified Data Engineer Associate or Professional certification.
• Bachelor's degree in Computer Science, Information Technology, Data Engineering, or a related field, or equivalent professional experience.
• Experience within the rental, transportation, fleet, logistics, automotive, manufacturing, or connected‑device industries is a plus.

What You'll Bring

The successful candidate will be more than a data pipeline developer. You'll bring the technical depth to work hands‑on with complex IoT data while also providing architectural direction and helping shape how the organization approaches IoT data for years to come.

You are a hands‑on technical leader who enjoys solving complex data challenges, mentoring engineers, collaborating with stakeholders, and turning large volumes of machine‑generated data into solutions that drive meaningful business outcomes.

Why This Opportunity?

• Own the direction of IoT data engineering within a growing enterprise environment.
• Help design a scalable, modern IoT data platform rather than simply implementing an established specification.
• Play a key role in the continued migration and expansion of IoT data within Databricks.
• Work at the intersection of IoT, data engineering, analytics, AI/ML, customer‑facing solutions, and sustainability.
• Provide technical leadership while remaining hands‑on with the technology.
• Join a collaborative data organization with opportunities to influence architecture, engineering standards, and future technology decisions.