Senior Big Data Engineer: Spark, Python & GCP Expert

Tech Genius inc; Phoenix (AZ); On-site; USD 120,000 - 160,000; Full time

Job summary

Amex is seeking an experienced Big Data Developer to design, develop, and optimize large-scale data pipelines using Spark and Python on Google Cloud Platform. You will build scalable ETL processes and collaborate with data engineers and business stakeholders to deliver reliable data solutions.

The role requires strong hands-on experience with Hadoop/Spark, PySpark, SQL, and GCP services such as BigQuery, Dataproc, Dataflow, Pub/Sub, and Cloud Storage.

Qualifications

• Bachelor’s or Master’s degree in Computer Science, Engineering, or related field.
• 5+ years of professional experience in Big Data development and data engineering.

Responsibilities

• Design and develop scalable data pipelines and ETL processes using Spark and Python.
• Work on Big Data ecosystems for handling large datasets across distributed systems.
• Deploy, manage, and optimize applications on Google Cloud Platform (GCP) services like BigQuery, Dataproc, Dataflow, and Pub/Sub.
• Collaborate with business stakeholders and data engineers to deliver high-quality data solutions.
• Ensure data quality, reliability, and security across applications.
• Participate in Agile/Scrum ceremonies and contribute to sprint deliverables.
• Troubleshoot and resolve performance issues in data pipelines and distributed systems.

Skills

Spark

PySpark

Python

GCP

Hadoop

DataFrames

SQL

CI/CD

Git

Jenkins

Education

Bachelor's/Master's in CS/Engineering

Tools

BigQuery

Dataproc

Dataflow

Pub/Sub

Cloud Storage

Job description

Amex is seeking an experienced Big Data Developer to design, develop, and optimize large-scale data pipelines using Spark and Python on Google Cloud Platform. You will build scalable ETL processes and collaborate with data engineers and business stakeholders to deliver reliable data solutions.

The role requires strong hands-on experience with Hadoop/Spark, PySpark, SQL, and GCP services such as BigQuery, Dataproc, Dataflow, Pub/Sub, and Cloud Storage.