Big Data Developer With GCP

Tech Genius inc; Phoenix (AZ); On-site; USD 120,000 - 160,000; Full time

Job summary

Amex is seeking an experienced Big Data Developer to design, develop, and optimize large-scale data pipelines using Spark and Python on Google Cloud Platform. You will build scalable ETL processes and collaborate with data engineers and business stakeholders to deliver reliable data solutions.

The role requires strong hands-on experience with Hadoop/Spark, PySpark, SQL, and GCP services such as BigQuery, Dataproc, Dataflow, Pub/Sub, and Cloud Storage.

Qualifications

• Bachelor’s or Master’s degree in Computer Science, Engineering, or related field.
• 5+ years of professional experience in Big Data development and data engineering.

Responsibilities

• Design and develop scalable data pipelines and ETL processes using Spark and Python.
• Work on Big Data ecosystems for handling large datasets across distributed systems.
• Deploy, manage, and optimize applications on Google Cloud Platform (GCP) services like BigQuery, Dataproc, Dataflow, and Pub/Sub.
• Collaborate with business stakeholders and data engineers to deliver high-quality data solutions.
• Ensure data quality, reliability, and security across applications.
• Participate in Agile/Scrum ceremonies and contribute to sprint deliverables.
• Troubleshoot and resolve performance issues in data pipelines and distributed systems.

Skills

Spark

PySpark

Python

GCP

Hadoop

DataFrames

SQL

CI/CD

Git

Jenkins

Education

Bachelor's/Master's in CS/Engineering

Tools

BigQuery

Dataproc

Dataflow

Pub/Sub

Cloud Storage

Job description

• We are seeking an experienced Big Data Developer to join our client Amex. The ideal candidate will have strong expertise in Big Data technologies, Spark, Python, and GCP (Google Cloud Platform) to design, develop, and optimize large-scale data processing pipelines and enterprise applications.
• Key Responsibilities
• Design and develop scalable data pipelines and ETL processes using Spark and Python.
• Work on Big Data ecosystems for handling large datasets across distributed systems.
• Deploy, manage, and optimize applications on Google Cloud Platform (GCP) services like BigQuery, Dataproc, Dataflow, and Pub/Sub.
• Collaborate with business stakeholders and data engineers to deliver high-quality data solutions.
• Ensure data quality, reliability, and security across applications.
• Participate in Agile/Scrum ceremonies and contribute to sprint deliverables.
• Troubleshoot and resolve performance issues in data pipelines and distributed systems.
• Required Skills
• Strong hands-on experience in Big Data technologies (Hadoop/Spark ecosystem).
• Proficiency in Apache Spark (PySpark, DataFrames, RDDs).
• Strong Python programming skills for data engineering and automation.
• Hands-on experience with GCP services (BigQuery, Dataflow, Dataproc, Pub/Sub, Cloud Storage).
• Solid understanding of ETL frameworks, data ingestion, and data transformation.
• Knowledge of SQL and database optimization techniques.
• Experience with CI/CD pipelines, Git, Jenkins is a plus.
• Good problem-solving and communication skills.

Qualifications

Bachelor’s or Master’s degree in Computer Science, Engineering, or related field.

5+ years of professional experience in Big Data development and data engineering.

• Prior experience in financial domain projects is a plus