Lead Data Engineer: Cloud, ML & Big Data

Capital One Group; New York (NY); On-site; USD 215,000 - 246,000; Full time

Job summary

Capital One is seeking a Lead Data Engineer in New York, NY, to drive big data initiatives and cloud-based data solutions. You will collaborate with cross-functional teams to design, implement, test, and optimize data pipelines, ensuring scalable and reliable performance.

The role requires strong programming skills (Python/SQL/Java/Scala), experience with distributed data tools, and a track record of real-time streaming work. Competitive salary and benefits offered.

Qualifications

• Bachelor's degree required.
• 4+ years in application development.
• 2+ years in big data technologies.
• 1+ year of cloud experience (AWS, Azure, Google).
• 7+ years in application development (preferred).
• 4+ years cloud experience (preferred).
• 4+ years distributed data/compute tools (MapReduce, Hadoop, Hive, EMR, Kafka, Spark, MySQL).
• 4+ years working on real-time data and streaming applications.
• 4+ years NoSQL experience (Mongo, Cassandra).
• 4+ years data warehousing experience (Redshift or Snowflake).
• 4+ years UNIX/Linux experience and shell scripting.
• 2+ years Agile engineering practices.

Responsibilities

• Collaborate with Agile teams to design, develop, test, and support technical solutions.
• Work with teams skilled in ML, distributed microservices, and full stack systems.
• Use Java, Scala, Python and SQL with cloud data warehousing services.
• Mentor other engineers and stay current with tech trends.
• Deliver cloud-based solutions that empower millions of users.

Skills

Python

SQL

Scala

Java

Big data

Cloud computing

Streaming data

NoSQL

Data warehousing

UNIX/Linux

Agile practices

Education

Bachelor's degree

Tools

AWS

Microsoft Azure

Google Cloud

Redshift

Snowflake

Hadoop

Hive

EMR

Kafka

Spark

MongoDB

Cassandra

UNIX/Shell

Job description

Capital One is seeking a Lead Data Engineer in New York, NY, to drive big data initiatives and cloud-based data solutions. You will collaborate with cross-functional teams to design, implement, test, and optimize data pipelines, ensuring scalable and reliable performance.

The role requires strong programming skills (Python/SQL/Java/Scala), experience with distributed data tools, and a track record of real-time streaming work. Competitive salary and benefits offered.