Remote Databricks Data Engineer — IoT Medallion Pipelines

Coherent Solutions; Town of Poland (NY); Hybrid; USD 110,000 - 155,000; Full time

Benefits offered by this job

Health insurance

Flexible remote/hybrid options

Referral program

Work anniversary program

Additional vacation days

Sports activities

Job summary

Coherent Solutions seeks an experienced Data Engineer to join a team building modern cloud-based data platforms. You will design and maintain scalable pipelines using Databricks, PySpark, and Python, with a focus on Medallion Architecture (Bronze/Silver/Gold) and API integrations for multi-source data ingestion.

The role emphasizes real-time and batch processing, data quality, and observability, collaborating with architects and product teams to deliver reliable analytics capabilities.

Qualifications

• Bachelor's degree or higher in CS or related field.
• 5+ years of Data Engineering experience.
• Strong Databricks experience and cloud data engineering skills.
• Proficiency in Python, PySpark, and SQL.
• Experience building data pipelines using Medallion Architecture.

Responsibilities

• Design, build, and maintain scalable data pipelines using Databricks.
• Develop API integrations to ingest data from multiple systems.
• Implement Medallion Architecture (Bronze, Silver, Gold) for analytics.
• Build batch and real-time data processing pipelines for performance.
• Create reusable data transformation frameworks in Python and PySpark.
• Optimize Spark jobs and Databricks workloads for cost efficiency.
• Ensure data quality with validation, monitoring, and tests.
• Collaborate with architects, product owners, and engineers.
• Troubleshoot production issues and improve pipeline reliability.
• Document data flows and architectural decisions.

Skills

Databricks

Python

PySpark

SQL

REST APIs

Git

Databricks Workflows

Medallion Architecture

ETL/ELT

Data quality

English communication

Education

Bachelor's degree in Computer Science or related field

Tools

Databricks

REST APIs

Git

SQL

Job description

Coherent Solutions seeks an experienced Data Engineer to join a team building modern cloud-based data platforms. You will design and maintain scalable pipelines using Databricks, PySpark, and Python, with a focus on Medallion Architecture (Bronze/Silver/Gold) and API integrations for multi-source data ingestion.

The role emphasizes real-time and batch processing, data quality, and observability, collaborating with architects and product teams to deliver reliable analytics capabilities.