DataBricks Developer (IoT sphere)

Coherent Solutions, Inc.; Oak Brook (IL); On-site; USD 110,000 - 170,000; Full time

Benefits offered by this job

Health insurance

Flexible work options

Job summary

Coherent Solutions, Inc. is seeking a Data Engineer for the IoT sphere to join a team building cloud-based data platforms with Databricks, Python, and PySpark. You will design scalable data pipelines and ingest data via REST APIs, enabling reliable analytics across devices.

You will implement Medallion Architecture (Bronze, Silver, Gold), optimize Spark workloads, and collaborate with architects and product teams to deliver robust, scalable data solutions in a hybrid, remote-friendly setting.

Qualifications

• 5+ years of experience in Data Engineering.
• Strong experience with Databricks.
• Experience designing data pipelines based on the Medallion Architecture (Bronze, Silver, Gold).
• Experience building scalable ETL/ELT data pipelines.
• Experience integrating data from REST APIs and third-party services.
• Experience developing real-time or near-real-time data pipelines.
• Experience optimizing Spark workloads for performance and scalability.
• Understanding of data quality, governance, and monitoring best practices.
• English proficiency at B2 level or higher.
• Bachelor's degree in Computer Science or a related field.

Responsibilities

• Design, build, and maintain scalable data pipelines using Databricks.
• Design and implement API integrations to ingest data from multiple systems.
• Implement and maintain Medallion Architecture-based pipelines for analytics.
• Build and optimize batch and real-time data processing to meet latency and reliability goals.
• Develop reusable data transformation frameworks with Python and PySpark.
• Optimize Spark jobs and Databricks workloads for performance and cost.
• Ensure data quality through validation, monitoring, and automated testing.
• Collaborate with architects, product owners, and engineering teams.
• Troubleshoot production issues and improve pipeline reliability.
• Document data flows, architectural decisions, and procedures.

Skills

Python

PySpark

SQL

REST APIs

Git

Apache Kafka

Education

Bachelor's degree in Computer Science or related field

Tools

Databricks

Delta Lake

Databricks Workflows

Unity Catalog

Apache Kafka

Job description

# DataBricks Developer (IoT sphere)## Locations* Bulgaria* Georgia* Lithuania* Mexico* Moldova* Poland* Ukraine## Company BackgroundOur client is a global leader in access solutions and smart home technologies. With over a decade of innovation in the IoT space, the company is a pioneer in connected home solutions, including garage door openers, gate systems, and smart access management tools. One of their flagship innovations is the platform, which allows users to control and monitor their home access systems through connected devices. Headquartered in Oak Brook, Illinois, our client serves both residential and commercial customers worldwide.## Project DescriptionThe specialist will join a team building modern cloud-based data platforms using Databricks, Python, and PySpark. The role focuses on designing and implementing scalable data pipelines based on the Medallion Architecture (Bronze, Silver, and Gold), as well as developing low-latency data services that integrate with multiple external systems through APIs.## Technologies* Python* PySpark* Databricks* SQL* REST APIs* Git* Delta Lake* Databricks Workflows* Apache Kafka* AWS / Azure## What You'll Do* Design, build, and maintain scalable data pipelines using Databricks;* Design and implement API integrations to ingest data from multiple internal and external systems;* Implement and maintain data pipelines based on the Medallion Architecture (Bronze, Silver, and Gold) to support reliable and scalable analytics;* Build and optimize batch and real-time data processing pipelines to meet performance, latency, and reliability requirements;* Develop reusable data transformation frameworks using Python and PySpark;* Optimize Spark jobs and Databricks workloads for performance and cost efficiency;* Ensure data quality through validation, monitoring, and automated testing;* Collaborate with architects, product owners, and engineering teams to design scalable data solutions;* Troubleshoot production issues and continuously improve pipeline reliability and observability;* Document data flows, architectural decisions, and operational procedures;## Job Requirements* 5+ years of experience in Data Engineering;* Strong experience with Databricks;* Advanced proficiency in Python, PySpark and storing SQL skills;* Experience designing and implementing data pipelines based on the Medallion Architecture (Bronze, Silver, and Gold);* Experience building scalable ETL/ELT data pipelines;* Experience integrating data from REST APIs and third-party services;* Experience developing real-time or near-real-time data pipelines;* Experience optimizing Spark workloads for performance and scalability;* Understanding of data quality, governance, and monitoring best practices;* Strong problem-solving and communication skills, with English proficiency at B2 level or higher;* Bachelor's degree in Computer Science or a related field;## Nice to Have* Experience with Delta Lake and Delta Live Tables (DLT);* Experience with Databricks Workflows;* Experience with Unity Catalog;* Experience with Apache Kafka or other streaming technologies;* Experience with Azure Data Factory, AWS Glue, or similar data orchestration tools;* Experience implementing CI/CD processes for data pipelines;* Experience working in Agile teams;## What Do We OfferThe global benefits package includes:* Technical and non-technical training for professional and personal growth;* Internal conferences and meetups to learn from industry experts;* Support and mentorship from an experienced employee to help you professional grow and development;* Health insurance;* Sports activities to promote a healthy lifestyle;* Flexible work options, including remote and hybrid opportunities;* Referral program for bringing in new talent;* Work anniversary program and additional vacation days.