Cloud Data Developer / Engineer

SGS Consulting; New York (NY); On-site; USD 110,000 - 170,000; Full time

Job summary

SGS Consulting is seeking a data engineer/cloud developer to design and implement scalable data architectures on AWS. You will work with big data and ML tools, build robust ETL pipelines, and help catalog data in data lakes.

You will leverage PySpark/Scala, handle imaging data like DICOMs, implement dashboards, and collaborate with IT teams to ensure security and compliance across cloud-based platforms.

Qualifications

• Experience with AWS services for big data and ML solutions.
• Familiarity with AWS EMR, AWS Glue, and Databricks.
• Must be familiar with cataloging data in big data scenarios such as data lakes utilizing tools like Hive or Glue Catalog.
• Must be familiar with creating data pipelines for big data ingestion.
• Experience with Python with PySpark, Scala, R, or SQL (Scala is a nice to have but not required).
• Ability to work with imaging files like DICOMs and how to extract metadata from those images and catalog the data.
• Experience with data cleansing and processing SAS datasets.
• Experience with sophisticated deep learning data science tools like Keras, PyTorch, and TensorFlow.
• Advanced data analysis
• Data storytelling
• Visual analytics with tools like Tableau, Spotfire, or QuickSight
• Must have deep expertise in ETL including the creation of destination files like ORC and Parquet.
• SAS experience preferred, but not required
• Experience with AWS and all services required for providing big data and machine learning solutions.
• Familiarity with big data platforms like AWS EMR, AWS Glue, and Databricks.
• Must be familiar with cataloging data in big data scenarios such as data lakes utilizing tools like Hive or Glue Catalog.
• Must be familiar with creating data pipelines for big data ingestion.
• Experience with Python with PySpark, Scala, R, or SQL (Scala is a nice to have but not required).
• Ability to work with imaging files like DICOMs and how to extract metadata from those images and catalog the data.
• Experience with data cleansing and processing SAS datasets
• Experience with sophisticated deep learning data science tools like Keras, PyTorch, and TensorFlow
• Advanced data analysis
• Data storytelling
• Visual analytics with tools like Tableau, Spotfire, or QuickSight
• Must have deep expertise in ETL including the creation of destination files like ORC and Parquet.
• SAS experience preferred, but not required

Responsibilities

• Provide experience with AWS and all services required for providing big data and machine learning solutions.
• Familiarity with big data platforms like AWS EMR, AWS Glue, and Databricks.
• Catalog data in big data scenarios such as data lakes utilizing tools like Hive or Glue Catalog.
• Create data pipelines for big data ingestion.
• Work with Python with PySpark, Scala, R, or SQL (Scala as nice to have).
• Work with imaging files like DICOMs and extract metadata and catalog the data.
• Data cleansing and processing SAS datasets.
• Use deep learning tools like Keras, PyTorch, and TensorFlow.
• Perform advanced data analysis and data storytelling.
• Create visual analytics with Tableau, Spotfire, or QuickSight.
• ETL expertise including the creation of destination files like ORC and Parquet.
• SAS experience preferred but not required.
• Collaborate on AWS data/ML projects.
• Coordinate with data engineering teams to maintain secure, compliant cloud environments.
• Develop and deploy cloud-based tools using Spark.

Skills

Big Data Analytics

Data Storytelling

Visual Analytics

ETL

Python / PySpark

DICOM Metadata Extraction

Tools

AWS

Databricks

Hive/Glue Catalog

Parquet/ORC

Job description

Job Responsibilities:

• Experience with AWS and all services required for providing big data and machine learning solutions


• Familiarity with big data platforms like AWS EMR, AWS Glue, and Databricks


• Must be familiar with cataloging data in big data scenarios such as data lakes utilizing tools like Hive or Glue Catalog.


• Must be familiar with creating data pipelines for big data ingestion


• Experience with Python with PySpark, Scala, R, or SQL (Scala is a nice to have but not required)


• Ability to work with imaging files like DICOMs and how to extract metadata from those images and catalog the data.


• Experience with data cleansing and processing SAS datasets


• Experience with sophisticated deep learning data science tools like Keras, PyTorch, and TensorFlow


• Advanced data analysis


• Data storytelling


• Visual analytics with tools like Tableau, Spotfire, or QuickSight


• Must have deep expertise in ETL including the creation of destination files like ORC and Parquet.


• SAS experience preferred, but not required


• Experience with AWS and all services required for providing big data and machine learning solutions


• Familiarity with big data platforms like AWS EMR, AWS Glue, and Databricks


• Must be familiar with cataloging data in big data scenarios such as data lakes utilizing tools like Hive or Glue Catalog.


• Must be familiar with creating data pipelines for big data ingestion


• Experience with Python with PySpark, Scala, R, or SQL (Scala is a nice to have but not required)


• Ability to work with imaging files like DICOMs and how to extract metadata from those images and catalog the data.


• Experience with data cleansing and processing SAS datasets


• Experience with sophisticated deep learning data science tools like Keras, PyTorch, and TensorFlow


• Advanced data analysis


• Data storytelling


• Visual analytics with tools like Tableau, Spotfire, or QuickSight


• Must have deep expertise in ETL including the creation of destination files like ORC and Parquet.


• SAS experience preferred, but not required

Skills:

Data Engineer / Cloud DeveloperAWSDatabricksData Engineer / Cloud DeveloperAWSDatabricks

Education/Experience:

• Proven track record of developing, deploying, and supporting data analytic tools


• Experience developing front-end interface to statistical models with tools like R/Shiny or others


• Experience managing and coordinating with IT teams to maintain secure and compliant tools and applications


• Experience with developing and deploying cloud-based tools or distributed computing environment using Spark


• Proven track record of developing, deploying, and supporting data analytic tools


• Experience developing front-end interface to statistical models with tools like R/Shiny or others


• Experience managing and coordinating with IT teams to maintain secure and compliant tools and applications


• Experience with developing and deploying cloud-based tools or distributed computing environment using Spark