AWS Data Migration Engineer - Big Data Pipelines & PySpark

RiseMe; Philadelphia (Philadelphia County); On-site; USD 120,000 - 180,000; Full time

Job summary

RiseMe is seeking a senior Big Data Engineer to accelerate data pipelines and analytics on AWS platform. You will design, migrate, automate, and monitor complex data flows using PySpark, Spark, and Databricks workflows, collaborating with cross-functional teams to meet requirements.

The role involves handling large real-world datasets, optimizing data ingestion from on-prem and cloud sources, and delivering scalable analytics solutions with an emphasis on performance and cost efficiency.

Qualifications

• 8+ years of software development experience.
• Experience with Big Data technologies on AWS.
• Design, migrate, automate data pipelines.
• Collaborate with cross-functional teams to translate requirements.

Responsibilities

• Support, migrate, automate, and monitor data pipelines.
• Ingest data from on-prem and cloud sources into aggregated data sets for reporting.
• Develop and deploy Big Data analytics solutions on AWS and Databricks.
• Monitor pipelines using PySpark and AWS services.

Skills

Big Data

PySpark

Python

Scala

AWS

Databricks

ETL pipelines

Data modeling

Education

Bachelor's degree in Computer Science

Tools

Databricks

AWS Glue

EMR

Step Functions

Lambda

Hadoop

Job description

RiseMe is seeking a senior Big Data Engineer to accelerate data pipelines and analytics on AWS platform. You will design, migrate, automate, and monitor complex data flows using PySpark, Spark, and Databricks workflows, collaborating with cross-functional teams to meet requirements.

The role involves handling large real-world datasets, optimizing data ingestion from on-prem and cloud sources, and delivering scalable analytics solutions with an emphasis on performance and cost efficiency.