Staff Security Data Engineer: Architect Exabyte-Scale Lakehouse

Adobe Inc.; San Jose (CA); On-site; USD 208,000 - 302,000; Full time

Job summary

Adobe's Security Data Platform team builds a petabyte-scale data lakehouse to enable threat detection, investigations, and security analytics. As a Staff Security Data Engineer, you will own end-to-end designs and lead critical initiatives across ingestion, storage, governance, and data serving.

This is a hands-on role focusing on production code, infrastructure, test strategy, and incident resolution. You will partner with security and infrastructure teams to deliver reliable, cost-effective

Qualifications

• Substantial experience building and operating production data platforms, typically 10 or more years in data or software engineering, with demonstrated Staff-level scope and impact.
• A record of personally delivering complex distributed systems and leading full-system design and implementation across team boundaries.
• Deep production expertise in Spark's traditional and real-time data workloads, Databricks, Delta Lake, and lakehouse architecture; strong Python and advanced SQL skills for data modeling, profiling, and query optimization.
• Strong AWS experience across storage, compute, networking, and IAM, plus production Terraform experience designing, testing, and maintaining reusable infrastructure modules.
• Practical understanding of distributed systems, including partitioning, replication, consistency, back pressure, scalability, and fault tolerance, with experience diagnosing failures across system boundaries.
• Experience with Airflow, MWAA, or equivalent orchestration, and with production observability, data quality, automated testing, CI/CD, and reliable deployment and recovery practices.
• Demonstrated ability to improve platform cost, performance, and reliability with measurable results; sound judgment in balancing these goals against security and consumer requirements.
• Effective technical communication and multi‑functional influence, including leading design reviews, resolving disagreements, mentoring engineers, and building adoption of shared engineering practices.

Responsibilities

• Own end-to-end designs that shape the Security Data Platform's fundamental architecture on Databricks and AWS, using Spark, Delta Lake, and Unity Catalog.
• Build and evolve distributed batch and streaming pipelines for EDR telemetry, network logs, identity events, cloud audit logs, and new security data sources.
• Write maintainable Python and SQL, develop complex platform features, and refactor unreliable code.
• Build and maintain tested, reusable Terraform modules for AWS and Databricks infrastructure, including S3, IAM, VPC networking, and Amazon MWAA.
• Lead diagnosis and resolution of complex production issues across Spark, storage, orchestration, and cloud infrastructure.

Skills

Spark

Databricks

Delta Lake

Python

SQL

AWS

Terraform

Airflow

MWAA

S3

Tools

IAM

VPC

Job description

Adobe's Security Data Platform team builds a petabyte-scale data lakehouse to enable threat detection, investigations, and security analytics. As a Staff Security Data Engineer, you will own end-to-end designs and lead critical initiatives across ingestion, storage, governance, and data serving.

This is a hands-on role focusing on production code, infrastructure, test strategy, and incident resolution. You will partner with security and infrastructure teams to deliver reliable, cost-effective