Data Engineer

SpyCloud, Inc.; Austin (TX); Híbrido; USD 120.000 - 150.000; Jornada completa

Ventajas ofrecidas por este puesto de trabajo

Health benefits

Remote-friendly culture

Competitive salaries and equity

401k matching

Descripción de la vacante

SpyCloud, Inc. is seeking a Data Engineer II to advance our data platform using Databricks and Spark. You will design and maintain scalable pipelines on AWS, including DynamoDB, Elasticsearch, and RDS-backed stores, enabling real-time and batch data delivery to customers.

You will own end-to-end data initiatives, collaborate with product and engineering teams, and help shape architecture and tooling. Strong Python development and ML/LLM data prep experience are valued.

Formación

• Experience with DynamoDB and AWS-native data services.
• Experience building Lakehouse/Data architectures in AWS.
• Strong Python development skills for data-intensive systems.
• 5+ years of related data/engineering experience.
• Experience troubleshooting data pipelines in production environments.
• Experience preparing data for ML/LLM workflows.
• Excellent communication and collaboration with stakeholders.
• Experience building ETL/data-processing pipelines.
• Experience with Elasticsearch/OpenSearch data stores.
• Terraform for infrastructure as code.
• Understanding data governance and privacy in AI/LLM environments.
• Observability/monitoring tools experience.
• Familiarity with AI/ML tools such as SageMaker LangChain or vector databases.

Responsabilidades

• Design, build, and optimize scalable data systems with a focus on performance and reliability.
• Develop and maintain real-time and batch data pipelines using AWS and Databricks.
• Own complex data engineering initiatives from design to deployment and production support.
• Collaborate with Product, SpyCloud Labs, and Engineering teams to meet requirements.
• Evaluate tooling for data processing, ML/AI workflows, and observability.
• Contribute to architectural decisions and implement scalable solutions.
• Promote engineering best practices through code reviews, documentation, and knowledge sharing.
• Address data quality, security, and reliability issues across production systems.

Conocimientos

Python programming

Data pipeline troubleshooting

Data governance awareness

Communication

Collaboration

Distributed systems understanding

AI/ML familiarity

Security/privacy awareness

Problem-solving

Incident response

Educación

Bachelor’s or Master’s degree in Computer Science, Engineering, or related STEM field

Herramientas

Databricks

Spark

Terraform

Elasticsearch/OpenSearch

DynamoDB

AWS

Go

C

Descripción del empleo

• This Data Engineer II role sits on the Engineering team, reporting to the Senior Director of Data Engineering.
• Your focus will include evolving our data platform using technologies such as Databricks and Spark, building scalable data pipelines, and serving customers through purpose-built data stores including RDS, Elasticsearch, and DynamoDB.
• This is a hands‑on individual contributor role with ownership of complex data engineering initiatives that support the scalability, reliability, and continued evolution of SpyCloud’s data platform
• Design, build, and improve scalable data systems with a focus on performance, and reliability
• Use Databricks and Spark to prepare, transform, and serve high-quality data to customers and downstream systems
• Build, optimize, and maintain real‑time and batch data pipelines using AWS technologies such as Lambda, S3, DynamoDB, ECS, EKS, and Terraform, as well as develop and maintain pipelines within the Databricks ecosystem
• Identify and implement improvements to the cost and performance of data ingestion, transformation, and storage workflows
• Be available for on‑call support to troubleshoot and remediate data pipeline failures outside of standard business hours
• Partner with Product Management, SpyCloud Labs, and Engineering teams to understand requirements and develop effective technical solutions
• Evaluate and implement modern tooling for data processing, ML/AI data workflows, and observability
• Contribute to architectural and technical design decisions and help implement scalable solutions aligned with engineering standards
• Promote engineering best practices through collaboration, code reviews, documentation, and knowledge sharing
• Identify and resolve complex data quality, performance, security, and reliability issues across production systems
• Independently own complex data engineering initiatives from design and development through deployment and production support
• Contribute to the continued evolution of SpyCloud’s data platform as product, customer, and AI requirements grow

Benefits

• Health benefits: From medical to dental, we’ve got you covered.
• Work/life balance: We offer generous PTO and a remote-friendly culture.
• Compensation: Our talented employees enjoy competitive salaries and equity.
• 401k matching: Investing in the future of our employees is a no‑brainer.
• SpyCares: We’re only as strong as the communities we’re a part of and giving back is in our DNA.

Practical experience with DynamoDB and other AWS-native data servicesExperience building and operating Lakehouse, Databricks, or large-scale distributed data architectures in AWSStrong development skills in Python, particularly when building data-intensive systems5+ years of professional experience in data engineering, software engineering, or a closely related disciplineExperience troubleshooting complex data pipeline, performance, and data quality issues in production environmentsExperience preparing and managing data for machine learning and LLM workflowsStrong communication and collaboration skills across technical and non-technical stakeholdersHands-on experience designing, building, and maintaining ETL and data-processing pipelinesExperience building and managing Elasticsearch/OpenSearch data storesExperience with infrastructure-as-code tools such as TerraformAbility to independently own complex technical projects from design and development through deployment and production supportStrong understanding of distributed systems as they relate to data storage, processing, and computingAbility to evaluate technical trade-offs and contribute effectively to architectural and design decisionsFamiliarity with AI/ML tools and technologies such as SageMaker, LangChain, or vector databasesUnderstanding of data governance, privacy, and compliance considerations within AI and LLM environmentsExperience with observability and monitoring tools for large-scale data platformsExperience with Go or C programming languagesComfort working in fast-paced environments with evolving product and technical requirementsExperience working with cybersecurity or security-related datasets and technologiesBachelor’s or Master’s degree in Computer Science, Engineering, or a related STEM field