Senior Data Engineer - Remote Real-Time Data & Spark

INSPYR Solutions; California (MO); On-site; USD 124,000 - 138,000; Full time

Job summary

INSPYR Solutions is seeking a Sr. Data Engineer for a long-term remote project in the US. The role focuses on building data products, ETL/ELT pipelines, and scalable analytics infrastructure in a cloud environment.

You will collaborate with product and engineering teams to shape the Live Analytics offering and deliver high-quality data systems. The candidate should have deep expertise in Spark, PySpark, Kafka, and SQL, with hands-on experience across data lakes/warehouses and AWS services.

Qualifications

• Bachelor's degree in CS or related field (not required).
• Experience building Data Lakes and Data Warehouses using big data and cloud stack.
• 5+ years of software development experience with Agile and DevOps for large-scale data engineering.
• Expert in Spark streaming and Spark concepts.
• Expert in Python and PySpark with 4+ years.
• Experience with Kafka streams and ingestion ETLs.
• Expert in ANSI SQL / Spark SQL with 5+ years.
• Experience with diverse data sources/formats (json, parquet, csv, delta).
• Experience with AWS and S3.
• Experience building ETL with batch and real-time streaming.
• Experience using Git and CI/CD.

Responsibilities

• Write new ETLs for real-time and feature data and reporting products.
• Gather requirements with product teams and design cost-aware solutions.
• Design peripheral systems: monitoring, ETL processes, offline workflows, deployment pipelines, metrics collection.
• Ensure best practices: testing, enterprise standards, target architecture, high availability, low latency.
• Create runbooks and documentation for systems and processes.
• Manage project delivery with daily stand-ups and stakeholder updates.

Skills

Spark streaming

Python

PySpark

Spark SQL

Kafka

ANSI SQL

AWS

ETL design

CI/CD

Data Lake/Data Warehouse

Education

Bachelor's degree in Computer Science or related field

Tools

Databricks

Git

Job description

INSPYR Solutions is seeking a Sr. Data Engineer for a long-term remote project in the US. The role focuses on building data products, ETL/ELT pipelines, and scalable analytics infrastructure in a cloud environment.

You will collaborate with product and engineering teams to shape the Live Analytics offering and deliver high-quality data systems. The candidate should have deep expertise in Spark, PySpark, Kafka, and SQL, with hands-on experience across data lakes/warehouses and AWS services.