Lead Cloud Platform Engineer: Spark & Airflow on AWS

Fidelity Investments Inc.; Westlake (TX); On-site; USD 180,000 - 240,000; Full time

Job summary

Fidelity Investments is seeking a hands-on Lead Cloud Platform Engineer to design, implement, and operate cloud-native infrastructure for large-scale data processing. You will own Spark data pipelines, Airflow DAGs, and a Kubernetes-based execution environment, ensuring reliable, observable, and secure platforms in AWS.

You will mentor engineers, drive CI/CD and observability practices, and help elevate reliability and performance across distributed systems.

Qualifications

• 8+ years in software, platform, or cloud engineering roles.
• Hands-on production experience with Apache Spark and Apache Airflow.
• Experience operating large-scale distributed systems in production.
• Strong communication and leadership skills.

Responsibilities

• Implement and operate cloud-native platform services for distributed data systems.
• Scale fault-tolerant, high-throughput systems aligned with architectural patterns.
• Own Spark data pipelines and Airflow orchestration layer and DAG execution.
• Tune Spark workloads and troubleshoot Spark jobs and Airflow DAGs.

Skills

Spark

Airflow

Kubernetes

AWS

Python/Java/Go

Leadership

Troubleshooting

Observability

CI/CD

Security

Education

Bachelor’s or Master’s in CS or related

Tools

Docker

GitHub Copilot

LLMs

AWS EKS

Terraform

Job description

Fidelity Investments is seeking a hands-on Lead Cloud Platform Engineer to design, implement, and operate cloud-native infrastructure for large-scale data processing. You will own Spark data pipelines, Airflow DAGs, and a Kubernetes-based execution environment, ensuring reliable, observable, and secure platforms in AWS.

You will mentor engineers, drive CI/CD and observability practices, and help elevate reliability and performance across distributed systems.