GCP Data Lead

Shrive Technologies; Charlotte (NC); On-site; USD 120,000 - 150,000; Full time

Job summary

Shrive Technologies LLC seeks a senior data engineer to design and optimize data pipelines on GCP, leveraging BigQuery, Airflow, PubSub, and Hadoop ecosystems.

You will build and optimize event-driven pipelines with Kafka and PubSub, implement scalable data models, and collaborate cross-functionally to advance banking and financial crime platforms using modern CI/CD practices.

Qualifications

• 5+ years of hands-on software development experience with Big Data & Analytics solutions.
• Experience with GCP Cloud technologies like BigQuery, Airflow, DataProc, PubSub.
• Strong SQL and data modeling for performance and scalability.
• Experience with event processing pipelines using Kafka or PubSub.
• Familiarity with CI/CD pipelines, automated tests, and source control tooling.

Responsibilities

• Design and optimize data pipelines for large-scale processing and analysis.
• Collaborate with product teams to deliver data-driven banking/financial crime features.
• Drive engineering practices, resilience, and automation in data platforms.

Tools

Airflow

PubSub

Kafka

Git

Jenkins

Hadoop

Hive

Spark

Python

Shell scripting

GitHub

Job description

JD

5+ years of hands-on software development experience with Big Data & Analytics solutions GCP Cloud - Big Query, Airflow, DataProc, PubSub, Hadoop Hive, Spark, Python, shell scripting Strong experience in designing, developing, and optimizing data pipelines for large-scale data processing, transformation, and analysis using Big Data and GCP technologies. Proficiency in SQL and database systems, with experience in designing and optimizing data models for performance and scalability. Experience of building event processing pipelines with Kafka or GCP PubSub. Design and development experience with Airflow, PubSub Kafka, Git, Jenkins is desirable. Knowledge of distributed (multi-tiered) systems, algorithms & relational databases. Strong Object-Oriented Programming skills and design patterns. Experience with CICD pipelines, Automated test frameworks, and source code management tools (XLR, Jenkins, Git). Good knowledge and experience with configuration management tools like GitHub Ability to analyze complex data engineering problems, propose effective solutions, and implement them effectively. Ability and willingness to learn, adopt and build solutions using the Enterprise frameworks established for Big Data Looks proactively beyond the obvious for continuous improvement opportunities. Communicates effectively with product and cross functional team. Willingness to learn new technologies and leverage them to their optimal potential. Understanding of various SDLC methodologies, familiarity with Agile & scrum ceremonies. Certifications in cloud platform (GCP Professional Data Engineer) is a plus. Provide engineering expertise to the team for developing features for Banking including Financial Crimes, Anti-money Laundering platforms Improve engineering practices, including driving resilience, predictability, and end efficiency through automation Partner with product teams to envision and deliver solutions which transform Financial Crimes Compliance using technology