Senior Data Engineer - Databricks & Feature Store

INSPYR Solutions; Town of Florida (NY); On-site; USD 140,000 - 190,000; Full time

Job summary

INSPYR Solutions in New York State is seeking a Senior Data Engineer to design, build, and maintain scalable data pipelines and analytics infrastructure. You will collaborate with data scientists and analysts to ensure reliable data for pricing optimization, model experimentation, and decision-making.

This role focuses on feature store expansion, data modeling, and ontology frameworks, while supporting data marketplace and writeback capabilities.

Qualifications

• Bachelor's degree in Computer Science, Engineering, or related field.
• 5+ years of experience in data engineering, ETL development, or a related role.
• Proficiency in SQL and Python, including Pandas, PySpark, and Python functions.

Responsibilities

• Design, build, and maintain robust real-time and batch data pipelines to ingest, transform, and deliver structured, semi-structured, and unstructured data from multiple sources.
• Develop and manage metadata-driven, real-time streaming pipelines and distributed data processing workflows at scale.
• Build and expand the feature store through data modeling, pipeline development, feature building, and data transforms that accelerate ML model readiness.
• Contribute to the design and build of pricing semantic layers, ontologies, and knowledge graphs to ensure consistent, reusable data definitions across the organization.
• Support Data Marketplace development, enabling teams to discover, access, and share data products across the organization.
• Develop cognitive search engine capabilities and support Data Syndication to distribute data across internal and external consumers.
• Build and expand the feature store through data modeling, pipeline development, feature building, and data transforms.
• Build data applications and support writeback capabilities that enable business users to interact with and act on data directly.
• Optimize database performance and query efficiency to support large-scale analytics workloads.
• Partner with cross-functional teams to understand data needs and translate them into scalable engineering solutions.

Skills

SQL

Python

Pandas

PySpark

Data Modeling

ETL Development

Cloud Data Platforms

dbt

Education

Bachelor's degree in Computer Science, Engineering, or related field

Tools

Databricks Unity Catalog

dbt

Job description

INSPYR Solutions in New York State is seeking a Senior Data Engineer to design, build, and maintain scalable data pipelines and analytics infrastructure. You will collaborate with data scientists and analysts to ensure reliable data for pricing optimization, model experimentation, and decision-making.

This role focuses on feature store expansion, data modeling, and ontology frameworks, while supporting data marketplace and writeback capabilities.