DataBricks Developer (Digital Engineering)

Coherent Solutions; Town of Poland (NY); Hybrid; USD 32,000 - 48,000; Full time

Benefits offered by this job

Health insurance

Remote/hybrid options

Conferences and meetups

Mentorship program

Vacation days

Job summary

Coherent Solutions is seeking a DataBricks Developer to support and extend a production Databricks-based data platform used for life insurance programs. You will collaborate with a small team to design data workflows, manage data access, and troubleshoot production pipelines.

The ideal candidate has 3+ years of data engineering experience with Databricks, SQL, Python, and Azure services, and enjoys working with stakeholders to deliver reliable data solutions.

Qualifications

• 3+ years of professional data engineering experience.
• Strong communication and collaboration with both technical and non-technical stakeholders.
• Hands-on Databricks development including Unity Catalog, Delta Lake, and notebooks.

Responsibilities

• Collaborate with a team of three engineers to develop and support data solutions in Databricks;
• Develop and maintain data processing workflows using Databricks, SQL, Python, and Delta Lake;
• Work with Unity Catalog to manage data access permissions, data lineage, and governance;
• Support the existing production system and participate in the implementation of new features and enhancements;
• Investigate and troubleshoot issues in Databricks data workflows and production processes;
• Collaborate with team members and stakeholders to deliver reliable data engineering solutions.

Skills

Databricks

SQL

Python

Azure

Team collaboration

Education

Bachelor’s degree in Computer Science or related field

Tools

Unity Catalog

Delta Lake

Databricks Genie

Job description

DataBricks Developer (Digital Engineering)

Poland

Company Background

Our client is a leading life insurance company and part of a global financial services group. Founded in 1896 and headquartered in North America, the company offers a wide range of products, including annuities and life insurance, through a nationwide network of agents.

Project Description

The project involves a production system for managing clients and the life insurance programs in which they are enrolled. The specialist will be responsible for supporting the existing system and developing new features.

Technologies

• Databricks, Unity Catalog, Delta Lake, Databricks Genie
• Advanced SQL
• Python

What You'll Do

• Collaborate with a team of three engineers to develop and support data solutions in Databricks;
• Develop and maintain data processing workflows using Databricks, SQL, Python, and Delta Lake;
• Work with Unity Catalog to manage data access permissions, data lineage, and governance;
• Support the existing production system and participate in the implementation of new features and enhancements;
• Investigate and troubleshoot issues in Databricks data workflows and production processes;
• Collaborate with team members and stakeholders to deliver reliable data engineering solutions;

Job Requirements

• 3+ years of professional experience in data engineering;
• Strong communication and collaboration skills, with the ability to work effectively with technical and non-technical stakeholders;
• Hands-on experience designing and developing data solutions in Databricks, including Databricks architecture, Unity Catalog, Delta tables, data lineage, access permissions, Databricks Genie, and notebooks;
• Advanced SQL skills, including writing complex queries and performing data transformations;
• Proficiency in Python for data processing, automation, and development of data engineering solutions;
• Professional experience working with Microsoft Azure cloud services and data platforms;

Nice To Have

• Experience using Databricks AI capabilities, including Agent Bricks, and implementing Generative AI and RAG solutions in Databricks;
• Experience working with Azure data services, including Azure Synapse, Synapse pipelines, Azure Data Factory, and Azure SQL pools;
• Experience setting up CI/CD processes for Databricks solutions using Azure DevOps, Databricks Asset Bundles, and Infrastructure as Code practices;
• Experience with performance tuning for Databricks workloads, SQL queries, and data processing pipelines;
• Databricks Certified Data Engineer Professional certification;

What Do We Offer

The global benefits package includes:

• Technical and non-technical training for professional and personal growth;
• Internal conferences and meetups to learn from industry experts;
• Support and mentorship from an experienced employee to help you professional grow and development;
• Health insurance;
• Sports activities to promote a healthy lifestyle;
• Flexible work options, including remote and hybrid opportunities;
• Referral program for bringing in new talent;
• Work anniversary program and additional vacation days.