Security Data Platform Engineer – Petabyte-Scale Cloud

Amazon; Herndon (VA); On-site; USD 143,700 - 194,400; Full time

Job summary

Amazon is seeking a software engineer for its Security Platform Engineering team to design, build, and operate a petabyte-scale data lake and governance services. You will drive the technical vision for data catalogs, optimization, and indexing using Iceberg, Parquet, and Spark-based engines.

You will contribute to distributed systems for low-latency threat detection, implement access controls, and collaborate with AWS and other business units.

Qualifications

• 3+ years of non-internship professional software development experience.
• 3+ years of non-internship design or architecture (design patterns, reliability and scaling) of new and existing systems experience.
• 3+ years of designing and developing large-scale, multi-tiered, multi-threaded, distributed software applications, tools, systems, and services using: C#, C++, Java, or Perl experience.
• 3+ years of Object Oriented Design experience.
• Bachelor's degree or foreign equivalent in Computer Science, Engineering, Mathematics, or a related field

Responsibilities

• Build and support Amazon Security's Data Lake, consisting of services for data cataloging, optimization, compaction, re-organization, and indexing at petabyte scale.
• Drive technical vision and contribute to the roadmap for systems that manage petabyte-scale data catalogs supporting Apache Iceberg, AWS Glue, Parquet, and other formats.
• Design and implement distributed systems for data optimization that balance multiple latency scenarios from minute-level batch detections to month-spanning threat hunting queries.
• Modernize security intelligence platform's data catalog and governance services to provide customers with rule-based access controls and fine-grained data access.
• Build and support query platform services for query scheduling, routing, auditing, and engine built on Apache Spark.
• Optimize data ingestion and cataloging systems that handle petabytes of data.
• Partner with data engineers and security teams across AWS, Amazon Stores, Digital, and other business units to understand requirements and deliver platform capabilities.
• Uphold software engineering best practices, code quality standards, and operational excellence.
• Explore and build AI driven data discovery capabilities using catalog metadata to help security analysts find relevant datasets faster.
• Work with AI/ML powered developer tools across the full SDLC, from AI assisted code reviews and design to agentic operational response

Skills

Java

Python

Scala

C++

Big data

Distributed systems

Multi-threading

Memory management

OO design

Communication

Education

Bachelor's degree in Computer Science/Engineering/Mathematics or related field

Tools

Apache Iceberg

Apache Spark

Trino

Flink

AWS Glue

Parquet

Job description

Amazon is seeking a software engineer for its Security Platform Engineering team to design, build, and operate a petabyte-scale data lake and governance services. You will drive the technical vision for data catalogs, optimization, and indexing using Iceberg, Parquet, and Spark-based engines.

You will contribute to distributed systems for low-latency threat detection, implement access controls, and collaborate with AWS and other business units.