Remote Senior Data Engineer - Azure Databricks & Pipelines

Hylant; Toledo (OH); Remote; USD 120,000 - 160,000; Full time

Job summary

Hylant is seeking a Data Engineer to build, integrate, and operationalize its enterprise data architecture on Databricks and Azure, translating designs into reliable, scalable data pipelines and analytics-ready datasets.

You will own CI/CD for data platforms, enforce governance through Unity Catalog, and deliver trusted datasets to Power BI and SQL workloads while collaborating with the Data & Digital Solutions team on agile deliveries.

Qualifications

• Bachelor's degree in computer science, engineering, data science, or related field, or equivalent practical experience.
• 5-7+ years as a Data Engineer designing, implementing, and validating production-grade data pipelines.
• Hands-on experience with Azure Databricks, Azure Data Factory, Delta Lake, Unity Catalog, and core Azure cloud services.
• Strong proficiency in SQL and PySpark for data transformation and pipeline development.
• Experience implementing data quality checks, reconciliations, and validation processes in complex data environments.
• Experience owning or contributing to CI/CD pipelines for data platforms (Azure DevOps Pipelines preferred).
• Ability to troubleshoot and resolve complex data, pipeline, and performance issues.
• Strong collaboration and communication skills, with the ability to bridge technical and non-technical stakeholders.
• Insurance industry experience preferred but not required.
• Exposure to PowerBI is a plus.

Responsibilities

• Design, build, and maintain data transformation logic, business rules, and analytics-ready datasets using Databricks notebooks, PySpark, and SQL.
• Develop, orchestrate, and monitor ETL/ELT pipelines using Azure Databricks and Azure Data Factory, including scheduling, dependency management, and job performance tuning.
• Create target schemas, curated tables, and analytics-ready data layers aligned to architectural designs and business requirements, including source-to-target mapping, historical data loads, and backfills for new data domains.
• Implement and enforce data governance, security, and access controls within Unity Catalog — including group-based grants, attribute-based access policies, and row- and column-level security — in accordance with standards defined by the Data Architecture team.
• Conduct data profiling, validation, reconciliation, and system integration testing to ensure end-to-end data quality, accuracy, and reliability; partner with source system owners and downstream consumers to investigate and resolve discrepancies.
• Serve curated, trusted datasets through Databricks SQL and Power BI datasets to support operational, financial, and analytical reporting.
• Own the CI/CD process for Databricks and Azure Data Factory using Azure DevOps Pipelines.
• Monitor, tune, and optimize the data platform for performance and cost, including cluster sizing, autoscaling, and job execution efficiency.
• Participate in agile delivery as part of the Data & Digital Solutions team, including sprint planning, backlog refinement, and delivery commitments.

Skills

SQL

PySpark

Data pipelines

Agile delivery

Education

Bachelor’s degree in computer science/engineering/data science

Tools

Azure Databricks

Azure Data Factory

Delta Lake

Unity Catalog

Azure DevOps Pipelines

Power BI

Job description

Hylant is seeking a Data Engineer to build, integrate, and operationalize its enterprise data architecture on Databricks and Azure, translating designs into reliable, scalable data pipelines and analytics-ready datasets.

You will own CI/CD for data platforms, enforce governance through Unity Catalog, and deliver trusted datasets to Power BI and SQL workloads while collaborating with the Data & Digital Solutions team on agile deliveries.