Senior Big Data Architect: Spark Streaming & GCP Migration

Jobright.ai; New York (NY); On-site; USD 140,000 - 190,000; Full time

Job summary

Accenture is seeking a Big Data Technology Architect in New York to design and implement scalable data processing solutions, lead migrations to Google Cloud Platform, and mentor junior engineers. The role focuses on optimizing Hadoop and Spark-based pipelines and aligning data architectures with business goals.

Responsibilities include developing backend services with Java Spring Boot, implementing GCP Dataproc and BigQuery, and creating reusable frameworks to accelerate delivery while

Qualifications

• Minimum 5 years experience in a Financial Institution.
• Minimum 5 years hands-on experience in Big Data, Hadoop & Data Analytics.
• Minimum 5 years hands-on experience in Java, Springboot, Microservices.
• Minimum 5 years experience in Data Warehouses.
• Minimum 2 years hands-on experience with Spark streaming.
• Bachelor's Degree or equivalent (12 years) work experience (If an, Associate's Degree with 6 years of work experience).
• Strategy, technology, operations, and AI professional services.
• Big Data Hadoop Java Spring Boot Spark Streaming Data Warehouses

Responsibilities

• Design and implement scalable, high-performance streaming and batch data processing solutions using Spark Streaming and related technologies.
• Develop backend applications and services using Java Spring Boot to support data pipelines and integrations.
• Write, review, and optimize code for data ingestion, transformation, and analytics.
• Lead the migration of Hadoop-based systems to GCP, ensuring seamless transition and minimal disruption.
• Implement GCP services such as Dataproc, BigQuery, and Cloud Storage for data processing and analytics.
• Develop and maintain tools to automate and streamline migration processes.
• Diagnose and resolve performance bottlenecks in Spark jobs, SQL queries, and streaming workflows.
• Optimize resource utilization for cloud-based and on-premises data systems.
• Analyze system logs and metrics to troubleshoot issues and ensure reliability.
• Work closely with cross-functional teams to align on technical requirements and deliverables.
• Provide technical guidance and mentorship to junior team members to improve their skills in Spark Streaming, Java Spring Boot, and Big Data tools.
• Architect Big Data solutions that are cost-effective, maintainable, and aligned with organizational goals.
• Develop reusable frameworks and libraries to accelerate project development and improve code quality.

Skills

Big Data

Data Analytics

Java

Spring Boot

Spark Streaming

Data Warehouses

GCP

Microservices

AWS

Education

Bachelor's Degree or equivalent (12 years) work experience

Tools

Hadoop

Spark

Google Cloud Platform

BigQuery

Dataproc

Job description

Accenture is seeking a Big Data Technology Architect in New York to design and implement scalable data processing solutions, lead migrations to Google Cloud Platform, and mentor junior engineers. The role focuses on optimizing Hadoop and Spark-based pipelines and aligning data architectures with business goals.

Responsibilities include developing backend services with Java Spring Boot, implementing GCP Dataproc and BigQuery, and creating reusable frameworks to accelerate delivery while