Data Platform Engineer II — Databricks on AWS

SpyCloud, Inc.; Austin (TX); Hybrid; USD 120,000 - 150,000; Full time

Benefits offered by this job

Health benefits

Remote-friendly culture

Competitive salaries and equity

401k matching

Job summary

SpyCloud, Inc. is seeking a Data Engineer II to advance our data platform using Databricks and Spark. You will design and maintain scalable pipelines on AWS, including DynamoDB, Elasticsearch, and RDS-backed stores, enabling real-time and batch data delivery to customers.

You will own end-to-end data initiatives, collaborate with product and engineering teams, and help shape architecture and tooling. Strong Python development and ML/LLM data prep experience are valued.

Qualifications

• Experience with DynamoDB and AWS-native data services.
• Experience building Lakehouse/Data architectures in AWS.
• Strong Python development skills for data-intensive systems.
• 5+ years of related data/engineering experience.
• Experience troubleshooting data pipelines in production environments.
• Experience preparing data for ML/LLM workflows.
• Excellent communication and collaboration with stakeholders.
• Experience building ETL/data-processing pipelines.
• Experience with Elasticsearch/OpenSearch data stores.
• Terraform for infrastructure as code.
• Understanding data governance and privacy in AI/LLM environments.
• Observability/monitoring tools experience.
• Familiarity with AI/ML tools such as SageMaker LangChain or vector databases.

Responsibilities

• Design, build, and optimize scalable data systems with a focus on performance and reliability.
• Develop and maintain real-time and batch data pipelines using AWS and Databricks.
• Own complex data engineering initiatives from design to deployment and production support.
• Collaborate with Product, SpyCloud Labs, and Engineering teams to meet requirements.
• Evaluate tooling for data processing, ML/AI workflows, and observability.
• Contribute to architectural decisions and implement scalable solutions.
• Promote engineering best practices through code reviews, documentation, and knowledge sharing.
• Address data quality, security, and reliability issues across production systems.

Skills

Python programming

Data pipeline troubleshooting

Data governance awareness

Communication

Collaboration

Distributed systems understanding

AI/ML familiarity

Security/privacy awareness

Problem-solving

Incident response

Education

Bachelor’s or Master’s degree in Computer Science, Engineering, or related STEM field

Tools

Databricks

Spark

Terraform

Elasticsearch/OpenSearch

DynamoDB

AWS

Go

C

Job description

SpyCloud, Inc. is seeking a Data Engineer II to advance our data platform using Databricks and Spark. You will design and maintain scalable pipelines on AWS, including DynamoDB, Elasticsearch, and RDS-backed stores, enabling real-time and batch data delivery to customers.

You will own end-to-end data initiatives, collaborate with product and engineering teams, and help shape architecture and tooling. Strong Python development and ML/LLM data prep experience are valued.