Streaming Data Platform Engineer

Bayside Solutions; United States; Teletrabalho; USD 76 000 - 90 000; Tempo integral

Resumo da oferta

Bayside Solutions, Inc. is seeking an experienced Streaming Data Platform Engineer to design and build large-scale, cloud-native data platform capabilities with a focus on real-time streaming.

You will create reusable services, SDKs, and frameworks to enable ML practitioners to process, manage, and access high-volume data reliably. You will work on distributed streaming systems, platform services, and self-service tooling, collaborating with platform teams and engineers to deliver scalable,

Qualificações

• Strong hands-on real-time streaming experience with Flink or Spark in production.
• Deep understanding of distributed data processing, state management, and recovery.
• Proficiency in Python or Java, API/service design, and automated tests.

Responsabilidades

• Design, build, and operate distributed streaming systems with Flink and/or Spark Structured Streaming.
• Develop production-grade platform services, APIs, and SDKs using Python/Java (Spring Boot).
• Contribute to control-plane and data-plane architecture, including orchestration and lifecycle management.
• Enable scalable data management for ML practitioners: datasets, indexing, and training-data workflows.
• Engineer observability, fault tolerance, checkpointing, and performance optimization for distributed systems.
• Collaborate with platform teams, application engineers, and ML practitioners to deliver self-service capabilities.

Conhecimentos

Streaming with Flink

Spark Streaming

Python/Java

API design

CI/CD

Distributed systems

Ferramentas

Databricks

Kafka

Kubernetes

Spring Boot

Descrição da oferta de emprego

Streaming Data Platform Engineer

W2 Contract

Pay Rate: $55 – $65 per hour

Location: Cupertino, CA – Remote Role

Job Summary:

We are seeking an experienced software engineer to help design and build large-scale, cloud-native data platform capabilities with a strong emphasis on real-time streaming. This is a platform engineering role, not a conventional ETL or analytics development position. The engineer will create reusable services, SDKs, frameworks, and self-service capabilities that enable internal consumers – including machine learning practitioners – to process, manage, and access high-volume data reliably.

Duties and Responsibilities:

• Design, build, and operate distributed streaming systems using Apache Flink and/or Apache Spark Structured Streaming for high-throughput, low-latency workloads.
• Develop production-grade platform services, APIs, reusable frameworks, and SDKs using Python and/or Java (Spring Boot).
• Contribute to control-plane and data-plane architecture, including workload orchestration, configuration, lifecycle management, and operational reliability.
• Enable scalable data management for ML consumers: dataset organization, efficient access and retrieval, indexing patterns, and repeatable training-data workflows.
• Engineer observability, fault tolerance, checkpointing, recovery, scaling, performance optimization, and automated testing for distributed processing systems.
• Partner with platform teams, application engineers, and ML practitioners to translate use cases into self-service capabilities and well-supported developer interfaces.
• Apply security and governance practices, including authentication, authorization, data protection, and appropriate handling of sensitive datasets.

Requirements and Qualifications:

• Strong hands-on real-time streaming experience with Apache Flink or Apache Spark Structured Streaming in production; able to explain design tradeoffs and incidents.
• Deep understanding of distributed data processing, partitioning, state management, event-time processing, backpressure, and failure recovery.
• Strong software engineering proficiency in Python or Java, including API/service design, automated tests, and maintainable code.
• Experience building data platform capabilities, internal developer platforms, reusable libraries/SDKs, or large-scale backend systems – beyond individual pipelines.
• Experience with microservices, containers, cloud-native deployment, CI/CD, monitoring, and production operations.
• Ability to work across requirements ambiguity, evaluate architecture options, and communicate tradeoffs with engineering stakeholders.
• Hands-on knowledge of both Flink and Spark, plus Databricks or comparable managed distributed-data platforms.
• Experience with Java/Spring Boot and Python in the same platform ecosystem.
• Experience supporting ML practitioners through training-data access, feature/dataset management, data quality, indexing, or governed data interfaces.
• Familiarity with Kafka or comparable event-streaming infrastructure; schema evolution and exactly-once/at-least-once delivery tradeoffs.
• Experience with Kubernetes, service orchestration, cloud object storage, data access governance, and RBAC/ABAC patterns.

Bayside Solutions, Inc. is not able to sponsor any candidates at this time. Additionally, candidates for this position must qualify as a W2 candidate.

Bayside Solutions, Inc. may collect your personal information during the position application process. Please reference Bayside Solutions, Inc.’s CCPA Privacy Policy at