Lead Data Engineer: Cloud, Spark & GenAI Innovator

Capital One; New York (NY); On-site; USD 215,200 - 245,600; Full time

Job summary

Capital One in New York, NY seeks a Lead Data Engineer (Python, AWS, Spark, Kafka, SQL, Snowflake, Databricks) to drive data pipelines and next-generation analytics for the Marketing and Messaging team.

You'll mentor engineers, design scalable data movement solutions, work across Agile teams, and implement robust data models, governance, and performance tuning using cloud-native data warehousing and streaming technologies.

Qualifications

• Bachelor's degree required.
• 4+ years in application development (internships not counted).
• 2+ years in big data technologies.
• 1 year of cloud computing experience (AWS/Azure/GCP).
• Master's degree preferred.
• 7+ years in development including Python, SQL, Spark, ETL tools or AWS Glue.

Responsibilities

• Collaborate with Agile teams to design, develop, test, and support data solutions.
• Lead developers with data movement, distributed computing, and full-stack systems experience.
• Utilize Python, SQL, and cloud data warehousing (Snowflake, Databricks) technologies.
• Mentor engineers and contribute to tech communities while improving performance.

Skills

Python

SQL

Spark

Kafka

ETL tools

AWS Glue

Education

Bachelor's degree

Master's degree

Tools

Snowflake

Databricks

Hadoop

Hive

EMR

MySQL

Job description

Capital One in New York, NY seeks a Lead Data Engineer (Python, AWS, Spark, Kafka, SQL, Snowflake, Databricks) to drive data pipelines and next-generation analytics for the Marketing and Messaging team.

You'll mentor engineers, design scalable data movement solutions, work across Agile teams, and implement robust data models, governance, and performance tuning using cloud-native data warehousing and streaming technologies.