IoT Data Engineer: Databricks & Real-Time Pipelines Remote

Coherent Solutions, Inc.; Oak Brook (IL); Hybrid; USD 110,000 - 170,000; Full time

Benefits offered by this job

Health insurance

Flexible work options

Job summary

Coherent Solutions, Inc. is seeking a Data Engineer for the IoT sphere to join a team building cloud-based data platforms with Databricks, Python, and PySpark. You will design scalable data pipelines and ingest data via REST APIs, enabling reliable analytics across devices.

You will implement Medallion Architecture (Bronze, Silver, Gold), optimize Spark workloads, and collaborate with architects and product teams to deliver robust, scalable data solutions in a hybrid, remote-friendly setting.

Qualifications

• 5+ years of experience in Data Engineering.
• Strong experience with Databricks.
• Experience designing data pipelines based on the Medallion Architecture (Bronze, Silver, Gold).
• Experience building scalable ETL/ELT data pipelines.
• Experience integrating data from REST APIs and third-party services.
• Experience developing real-time or near-real-time data pipelines.
• Experience optimizing Spark workloads for performance and scalability.
• Understanding of data quality, governance, and monitoring best practices.
• English proficiency at B2 level or higher.
• Bachelor's degree in Computer Science or a related field.

Responsibilities

• Design, build, and maintain scalable data pipelines using Databricks.
• Design and implement API integrations to ingest data from multiple systems.
• Implement and maintain Medallion Architecture-based pipelines for analytics.
• Build and optimize batch and real-time data processing to meet latency and reliability goals.
• Develop reusable data transformation frameworks with Python and PySpark.
• Optimize Spark jobs and Databricks workloads for performance and cost.
• Ensure data quality through validation, monitoring, and automated testing.
• Collaborate with architects, product owners, and engineering teams.
• Troubleshoot production issues and improve pipeline reliability.
• Document data flows, architectural decisions, and procedures.

Skills

Python

PySpark

SQL

REST APIs

Git

Apache Kafka

Education

Bachelor's degree in Computer Science or related field

Tools

Databricks

Delta Lake

Databricks Workflows

Unity Catalog

Apache Kafka

Job description

Coherent Solutions, Inc. is seeking a Data Engineer for the IoT sphere to join a team building cloud-based data platforms with Databricks, Python, and PySpark. You will design scalable data pipelines and ingest data via REST APIs, enabling reliable analytics across devices.

You will implement Medallion Architecture (Bronze, Silver, Gold), optimize Spark workloads, and collaborate with architects and product teams to deliver robust, scalable data solutions in a hybrid, remote-friendly setting.