Lead Data Engineer — Python, AWS, Spark & GenAI

Capital One National Association; United States; On-site; USD 197,000 - 225,000; Full time

Job summary

Capital One is seeking a Lead Data Engineer to design and deliver cloud-based data solutions using Python, SQL, Spark, and AWS services. You’ll work with cross-functional teams to build scalable data pipelines, optimize data models, and mentor engineers while driving performance. Experience with Snowflake, Databricks, and real-time streaming is highly valued.

The role supports Capital One’s data initiatives across multiple U.S. locations; competitive salary and comprehensive benefits apply.

Qualifications

• Bachelor’s Degree is required.
• 4+ years in application development.
• 2+ years in big data technologies.
• 1+ year experience with cloud computing (AWS, Azure, GCP).
• Preferred: Master’s Degree.
• 7+ years of experience in development including Python, SQL, Spark, ETL tools, or AWS Glue.
• 4+ years with a public cloud (AWS, Azure, GCP).
• 4+ years with distributed data/compute tools (MapReduce, Hadoop, Hive, EMR, Kafka, Spark, MySQL).
• 4+ years working on real-time data and streaming apps.
• 4+ years NoSQL experience (Mongo, Cassandra).
• 4+ years data warehousing experience (Redshift or Snowflake).
• 4+ years UNIX/Linux experience and shell scripting.
• 4+ years data warehousing modeling.
• 2+ years Agile engineering practices.
• Experience leveraging interactive AI tooling to accelerate delivery.

Responsibilities

• Collaborate with Agile teams to design, develop, test, implement, and support data solutions.
• Lead developers with expertise in data movement, distributed computing, and full-stack systems.
• Utilize Python, SQL, Snowflake and Databricks in cloud data warehousing.
• Optimize information systems for end-users and downstream consumers.
• Mentor engineers and stay current with technology trends and communities.
• Collaborate with product managers to deliver cloud-based data experiences for customers.
• Perform unit tests and code reviews to ensure well-designed, efficient code.

Education

Bachelor’s Degree

Tools

Python

SQL

Spark

Kafka

Snowflake

Databricks

AWS

GenAI

Job description

Capital One is seeking a Lead Data Engineer to design and deliver cloud-based data solutions using Python, SQL, Spark, and AWS services. You’ll work with cross-functional teams to build scalable data pipelines, optimize data models, and mentor engineers while driving performance. Experience with Snowflake, Databricks, and real-time streaming is highly valued.

The role supports Capital One’s data initiatives across multiple U.S. locations; competitive salary and comprehensive benefits apply.