Data Engineer – III – Databrciks

INSPYR Solutions; Town of Florida (NY); On-site; USD 140,000 - 190,000; Full time

Job summary

INSPYR Solutions in New York State is seeking a Senior Data Engineer to design, build, and maintain scalable data pipelines and analytics infrastructure. You will collaborate with data scientists and analysts to ensure reliable data for pricing optimization, model experimentation, and decision-making.

This role focuses on feature store expansion, data modeling, and ontology frameworks, while supporting data marketplace and writeback capabilities.

Qualifications

• Bachelor's degree in Computer Science, Engineering, or related field.
• 5+ years of experience in data engineering, ETL development, or a related role.
• Proficiency in SQL and Python, including Pandas, PySpark, and Python functions.

Responsibilities

• Design, build, and maintain robust real-time and batch data pipelines to ingest, transform, and deliver structured, semi-structured, and unstructured data from multiple sources.
• Develop and manage metadata-driven, real-time streaming pipelines and distributed data processing workflows at scale.
• Build and expand the feature store through data modeling, pipeline development, feature building, and data transforms that accelerate ML model readiness.
• Contribute to the design and build of pricing semantic layers, ontologies, and knowledge graphs to ensure consistent, reusable data definitions across the organization.
• Support Data Marketplace development, enabling teams to discover, access, and share data products across the organization.
• Develop cognitive search engine capabilities and support Data Syndication to distribute data across internal and external consumers.
• Build and expand the feature store through data modeling, pipeline development, feature building, and data transforms.
• Build data applications and support writeback capabilities that enable business users to interact with and act on data directly.
• Optimize database performance and query efficiency to support large-scale analytics workloads.
• Partner with cross-functional teams to understand data needs and translate them into scalable engineering solutions.

Skills

SQL

Python

Pandas

PySpark

Data Modeling

ETL Development

Cloud Data Platforms

dbt

Education

Bachelor's degree in Computer Science, Engineering, or related field

Tools

Databricks Unity Catalog

dbt

Job description

Senior Data Engineer

Overview

The Data Engineer designs, builds, and maintains scalable data pipelines and infrastructure that power analytics, machine learning, and business intelligence initiatives. Working closely with Data Scientists, analysts, and business stakeholders, this role ensures that high-quality, reliable data is accessible across the organization to support pricing optimization, model experimentation, and strategic decision-making. A key focus of this role is accelerating feature store expansion through robust data modeling, pipeline development, and feature engineering, while contributing to pricing semantic and ontology frameworks that drive consistency across the data ecosystem.

Duties and Responsibilities

• Design, build, and maintain robust real-time and batch data pipelines to ingest, transform, and deliver structured, semi-structured, and unstructured data from multiple sources
• Develop and manage metadata-driven, real-time streaming pipelines and distributed data processing workflows at scale
• Build and expand the feature store through data modeling, pipeline development, feature building, and data transforms that accelerate ML model readiness
• Contribute to the design and build of pricing semantic layers, ontologies, and knowledge graphs to ensure consistent, reusable data definitions across the organization
• Support Data Marketplace development, enabling teams to discover, access, and share data products across the organization
• Develop cognitive search engine capabilities and support Data Syndication to distribute data across internal and external consumers
• Build and expand the feature store through data modeling, pipeline development, feature building, and data transforms
• Build data applications and support writeback capabilities that enable business users to interact with and act on data directly
• Optimize database performance and query efficiency to support large-scale analytics workloads
• Partner with cross-functional teams to understand data needs and translate them into scalable engineering solutions

Minimum Qualifications/Requirements

• Bachelor's degree in Computer Science, Engineering, or related field
• 5+ years of experience in data engineering, ETL development, or a related role
• Proficiency in SQL and Python, including Pandas, PySpark, and Python functions
• Experience with cloud data platforms (e.g., Databricks Unity Catalog) and pipeline orchestration tools (e.g. dbt)
• Demonstrated experience with data modeling, feature engineering, and building transforms for ML feature stores
• Strong understanding of data modeling, warehousing concepts, and relational databases

Preferred Skills

• Experience with data quality frameworks such as Great Expectations and Databricks Delta Sharing
• Working knowledge of Semantic Engineering, flexible data modeling, and ontology design (OWL, knowledge graphs)
• Familiarity with Data Governance frameworks, Master Data Management practices, and Data Migration methodologies
• Experience building pricing semantic layers or data ontologies to standardize business definitions across teams
• App building skills are a strong plus — experience developing data apps, dashboards, or writeback solutions (e.g., Streamlit, Hex, or similar tools)
• Experience supporting ML pipelines and working alongside Data Science teams
• Knowledge of DevOps practices including CI/CD and version control (Git)
• Strong communication skills and ability to work across technical and business teams
• Attention to detail and commitment to data accuracy and integrity

INSPYR Solutions provides Equal Employment Opportunities (EEO) to all employees and applicants for employment without regard to race, color, religion, sex, national origin, age, disability, genetics, or any other protected status. INSPYR Solutions complies with all applicable laws governing nondiscrimination in employment in every location in which the company has facilities.