Senior Data Engineer — Python, AWS, Spark, GenAI

Capital One; New York (NY); On-site; USD 215,000 - 246,000; Full time

Job summary

Capital One is hiring a Data Engineer to design and deliver scalable, cloud-first data platforms and real-time streaming pipelines. You’ll lead cross-functional teams, implement robust data governance, and work with Databricks/Snowflake to support omnichannel marketing data needs.

You will mentor developers and data analysts, champion best practices, and drive performance and reliability across large-scale data workloads. A strong multi-language skillset and cloud experience are essential.

Qualifications

• Bachelor’s degree in Computer Science or a related quantitative field.
• At least 4 years of experience in application development.
• At least 2 years of experience in distributed data.
• At least 2 years of experience with SQL.
• At least 2 years of experience with one programming language (Python, Java, or Scala).
• At least 2 years of experience in data pipeline design and development.
• At least 1 year of experience in data modeling and designing end-to-end data solutions using relational and non-relational databases.

Responsibilities

• Architect and build resilient, scalable, cloud-first data solutions, lakehouse architectures, and real-time streaming pipelines.
• Lead and influence cross-functional Agile teams; enforce data engineering design patterns for quality and maintainability.
• Leverage Python, SQL, Scala, Spark, and NoSQL/RDBMS alongside Databricks and Snowflake.
• Implement data security standards and uphold data quality, governance, and permissible use.
• Partner with product managers and engineers to deliver robust data solutions that meet business needs.
• Communicate technical concepts clearly to stakeholders and stay updated on tech trends.
• Perform unit tests, code reviews, and optimize systems for performance and observability.

Skills

Python

SQL

Scala

Data modeling

Data pipelines

Distributed data

Cloud platforms

Education

Bachelor's degree in Computer Science or related quantitative field

Master’s degree in Computer Science or related field

Tools

Databricks

Snowflake

EMR

Hadoop

Kafka

Airflow

Dagster

MongoDB

DynamoDB

Job description

Capital One is hiring a Data Engineer to design and deliver scalable, cloud-first data platforms and real-time streaming pipelines. You’ll lead cross-functional teams, implement robust data governance, and work with Databricks/Snowflake to support omnichannel marketing data needs.

You will mentor developers and data analysts, champion best practices, and drive performance and reliability across large-scale data workloads. A strong multi-language skillset and cloud experience are essential.