Senior Lead Data Engineer - Real-Time Analytics on Cloud

Capital One National Association; Plano (TX); On-site; USD 209,000 - 239,000; Full time

Job summary

Capital One is seeking a Senior Lead Data Engineer to lead the design and build of an enterprise-level streaming data platform. You will guide a team using Flink, Spark, Kafka and cloud data warehouses like Snowflake/Redshift, while driving best practices in data architecture and CI/CD in a collaborative, agile environment.

The role requires 6+ years in application development, 2+ years in big data, and 1 year in cloud computing (AWS/Azure/GCP).

Qualifications

• Bachelor’s degree required in software development.
• Minimum 6 years of application development experience.
• 2+ years of big data technologies and 1 year cloud experience.

Responsibilities

• Lead design and build Enterprise Level scalable, low-latency streaming data platform.
• Build the next generation Distributed Streaming Data Pipelines and Analytics Data Stores using streaming frameworks (Flink, Spark Streaming) using programming languages like Java, Scala, Python.
• Lead a group of engineers building data pipelines using big data technologies (Spark, Flink, Kafka, Snowflake, AWS Big Data Services, Snowflake, Redshift) on medium to large scale datasets.
• Influence best practices for Data Pipeline design, Data architecture and processing of structured and unstructured data.
• Work in a collaborative agile environment with CI/CD and security partnerships.
• Collaborate with Agile teams to design, develop, test, implement, and support technical solutions.
• Lead a team of developers with deep experience in ML, distributed microservices, and full stack systems.
• Utilize Java, Scala, Python and open source RDBMS/NoSQL databases and cloud data warehousing like Redshift and Snowflake.
• Share passion for staying on top of tech trends and mentoring engineers.
• Collaborate with product managers to deliver cloud-based solutions that empower users.

Skills

Java

Scala

Python

Spark

Flink

Kafka

Snowflake

Redshift

AWS

Cloud

CI/CD

Agile

Leadership

SQL

NoSQL

Mongo

Education

Bachelor’s Degree

Master's Degree

Tools

MapReduce

Hadoop

Hive

EMR

Kafka

Spark

MySQL

Mongo

Job description

Capital One is seeking a Senior Lead Data Engineer to lead the design and build of an enterprise-level streaming data platform. You will guide a team using Flink, Spark, Kafka and cloud data warehouses like Snowflake/Redshift, while driving best practices in data architecture and CI/CD in a collaborative, agile environment.

The role requires 6+ years in application development, 2+ years in big data, and 1 year in cloud computing (AWS/Azure/GCP).