Senior Data Engineer – Remote, AI-Driven Lakehouse Architect

SpyCloud; Austin (TX); Hybrid; USD 144,000 - 187,000; Full time

Benefits offered by this job

401(k) with Employer Contribution

Health, Vision, and Dental Insurance

HSA with Employer Contribution

Paid time off

16 paid holidays

Remote-friendly work options

Job summary

SpyCloud in Austin, TX seeks a Senior Data Engineer to lead the architectural transformation of our data platform, collaborating with Product Management, SpyCloud Labs, and Engineering teams to modernize data systems using Databricks and serving customers via RDS, Elasticsearch, and DynamoDB.

You will mentor data engineers, drive AI-ready data pipelines, and own long-term data architecture aligned with the company’s AI-driven goals and security-first practices.

Qualifications

• 8+ years of experience in data engineering or closely related disciplines.
• 2+ years in a technical leadership or senior engineering role.
• Experience building and operating Lakehouse, Databricks, or large-scale distributed data lake architectures in AWS.
• Knowledge of distributed systems as it pertains to data storage and computing.
• Experience building and managing large Elasticsearch/Opensearch data stores.
• Hands-on experience building ETL scripts.
• Hands-on experience preparing and managing data for machine learning and LLM workflows.
• Strong development skills in Python, especially in building data-intensive systems.
• Practical experience using DynamoDB and other AWS-native data services at scale.
• Effective communication and collaboration across technical and non-technical stakeholders.
• Proven ability to mentor and grow engineering talent and lead large-scale initiatives.

Responsibilities

• Redesign and modernize data architecture for scalability, performance, and AI-readiness.
• Use Databricks/Spark to prepare, transform, and serve data to customers.
• Build and optimize real-time and batch data pipelines using AWS tools (Lambda, S3, DynamoDB, ECS, EKS, Terraform).
• Design systems to prepare and serve high-quality data for ML models and LLM applications.
• Optimize cost and performance of daily data ingestion, transformation, and storage workflows.
• Work with Product and Engineering teams to ensure data solutions meet product requirements.
• Evaluate and implement modern tooling for data processing, ML data workflows, and observability.
• Guide and mentor data engineers, championing best practices and technical excellence.
• Ensure operational data quality, security, and reliability across systems.
• Lead long-term data architecture planning aligned with company and AI-driven goals.

Skills

Data engineering

Leadership

Python

Databricks

AWS

Elasticsearch

DynamoDB

ETL pipelines

Education

Bachelor's or Master's in Computer Science/Engineering

Tools

Databricks

Spark

AWS

Lambda

S3

DynamoDB

Elasticsearch/OpenSearch

Job description

SpyCloud in Austin, TX seeks a Senior Data Engineer to lead the architectural transformation of our data platform, collaborating with Product Management, SpyCloud Labs, and Engineering teams to modernize data systems using Databricks and serving customers via RDS, Elasticsearch, and DynamoDB.

You will mentor data engineers, drive AI-ready data pipelines, and own long-term data architecture aligned with the company’s AI-driven goals and security-first practices.