AWS Data Migration Engineer

RichRelevance; New York (NY, DE); On-site; USD 130,000 - 170,000; Full time

Job summary

RichRelevance seeks a senior Big Data engineer to integrate and optimize complex real-world customer TV and digital ad data for pre- and post-campaign analyses on AWS-based big data platforms. You will work with Databricks, PySpark, Glue, EMR, and Lambda to architect scalable pipelines and collaborate with cross-functional teams.

Requirements include 8+ years in software development, strong experience with Hadoop ecosystem, Scala, Python, and building monitored data workflows on cloud

Qualifications

• 8+ years of software development experience.
• Experience with Hadoop ecosystem and Scala, PySpark, Python.
• Gathers requirements, designs, supports, migrates, automates.
• Monitors data pipelines using PySpark and AWS services.

Responsibilities

• Support, migration, automation and monitoring of data pipelines.
• Ingest data from multiple sources and create aggregated data sets.
• Collaborate with cross-functional teams to meet reporting needs.
• Implement scalable,Cost-effective big data analytics on cloud platforms.

Skills

Big Data Technologies

Scala

PySpark

Python

AWS services

Data pipeline monitoring

Tools

Databricks

Glue

EMR

Step-functions

Lambda

Job description

To integrate complex real-world Customer TV and Digital Ad viewing data to perform pre- campaign planning and post-campaigns performance measurements using Big Data Platform (Databricks, Glue, EMR, Step-functions, Lambda, PySpark and other AWS services to support existing workloads, Automate and monitor and based on requirement develop an elastic, scalable, high performance and cost-effective solution in collaboration with cross functional teams.

Key Goals Specified by Manthan / Client(s)

• Position is responsible for the support, migration, automation and monitoring of existing data pipelines and based on new requirements, design, develop, test, and deploy the Big Data Analytics solutions on cloud (AWS) and Databricks.
• Collaborate with key stake holders and translate business requirements to technical requirements & implement solutions.
• Dive into large, noisy, and complex real-world customer TV and Digital Ad viewing data to do pre-campaign planning and post campaigns performance measurement using Big data Platform (Databricks, PySpark, Glue, EMR).
• Responsible for support, migration, automation and monitoring of data pipelines to ingest data, integrate data from multiple data sources (On-Premise & Cloud) and create aggregated data sets for reporting needs and appropriate notifications.

Job Description

• 8+ years of overall software development
• Experience on Big Data Technologies, Hadoop Eco system and Technologies with experience programming in Scala, PySpark and Python
• Requirement gathering, design and Support, Migration, Automation
• Monitoring of data pipelines with PySpark, AWS services

Location States ( NJ, NY, MA, CT, NC, PA preferred)