Senior Data Engineer - Remote Real-Time Data & Pipelines

INSPYR Solutions; California (MO); Hybrid; USD 124,000 - 138,000; Full time

Benefits offered by this job

Comprehensive medical benefits

Competitive pay

401(k) retirement plan

Job summary

INSPYR Solutions seeks a Senior Data Engineer for a long-term remote project in the United States. You will design, develop, and support data pipelines, ETL/ELT processes, and data products using Spark, Python, and cloud services.

You will collaborate with product and engineering teams to gather requirements, ensure cost efficiency, scalability, and high availability, delivering robust data solutions for Live Analytics offerings.

Qualifications

• Bachelor's degree in Computer Science or related field (not required to have a degree).
• Experience building Data Lakes and Data Warehouses on cloud stack.
• 5+ years of software development experience using Agile and DevOps for large-scale data engineering.
• Expert proficiency in Spark streaming and Spark concepts.
• Expert proficiency in Python and PySpark with 4+ years of experience.
• Experience working with Kafka streams and building ingestion ETLs.
• Expert proficiency in ANSI SQL / Spark SQL with 5+ years.
• Experience with diverse data sources and formats (json, parquet, csv, psv, delta).
• Experience with AWS and S3 cloud services.
• Experience building ETL with batch and real-time streaming.
• Experience using Git and CI/CD.

Responsibilities

• Write new ETLs for real-time and feature data and reporting products.
• Gather requirements with product teams and design cost-aware solutions.
• Design and develop monitoring, ETL processes, and automated workflows.
• Ensure best practices for test coverage, architecture, high availability, and performance.
• Document systems and runbooks for maintenance and troubleshooting.
• Manage day-to-day delivery with agile stand-ups and stakeholder communication.

Skills

Spark Streaming

Python

PySpark

Kafka

SQL / Spark SQL

AWS

Data Lake / Data Warehouse

Agile / DevOps

Education

Bachelor's degree in Computer Science or related field

Tools

Databricks

Git

CI/CD

Job description

INSPYR Solutions seeks a Senior Data Engineer for a long-term remote project in the United States. You will design, develop, and support data pipelines, ETL/ELT processes, and data products using Spark, Python, and cloud services.

You will collaborate with product and engineering teams to gather requirements, ensure cost efficiency, scalability, and high availability, delivering robust data solutions for Live Analytics offerings.