Data Engineer II: Scalable Data Pipelines for Cyber Defense

SpyCloud, Inc.; United States; On-site; USD 104,000 - 135,000; Full time

Benefits offered by this job

401(k) plan

Health/Dental/Vision insurance

HSA with employer contribution

Life/Disability insurance

PTO and holidays

Job summary

SpyCloud is seeking a Data Engineer II to join our Engineering team, reporting to the Senior Director of Data Engineering. You will evolve our data platform using Databricks and Spark, build scalable data pipelines, and serve customers via data stores like RDS, Elasticsearch, and DynamoDB.

This is a hands-on individual contributor role with ownership of complex data initiatives. You will design and implement scalable data systems, optimize real-time and batch pipelines on AWS (Lambda, S3,

Qualifications

• 5+ years of professional experience in data engineering, software engineering, or a closely related discipline.
• Experience building and operating Lakehouse, Databricks, or large-scale distributed data architectures in AWS.
• Strong understanding of distributed systems as they relate to data storage, processing, and computing.

Responsibilities

• Design, build, and improve scalable data systems with a focus on performance and reliability.
• Use Databricks and Spark to prepare, transform, and serve high-quality data to customers and downstream systems.
• Build, optimize, and maintain real-time and batch data pipelines using AWS technologies such as Lambda, S3, DynamoDB, ECS, EKS, and Terraform, as well as develop and maintain pipelines within the Databricks ecosystem.
• Identify and implement improvements to the cost and performance of data ingestion, transformation, and storage workflows.
• Be available for on-call support to troubleshoot and remediate data pipeline failures outside of standard business hours.
• Partner with Product Management, SpyCloud Labs, and Engineering teams to understand requirements and develop effective technical solutions.

Skills

Python

Databricks/Spark

AWS

ETL pipelines

Distributed systems

Education

Bachelor's degree in CS or related

Tools

Databricks ecosystem

Terraform

DynamoDB

Elasticsearch/OpenSearch

Job description

SpyCloud is seeking a Data Engineer II to join our Engineering team, reporting to the Senior Director of Data Engineering. You will evolve our data platform using Databricks and Spark, build scalable data pipelines, and serve customers via data stores like RDS, Elasticsearch, and DynamoDB.

This is a hands-on individual contributor role with ownership of complex data initiatives. You will design and implement scalable data systems, optimize real-time and batch pipelines on AWS (Lambda, S3,