Cloud Data Platform Engineer

Global Channel Management, Inc; New York (NY); Hybrid; USD 110,000 - 140,000; Full time

Job summary

A leading data solutions company is seeking a Cloud Data Platform Engineer to implement data applications on a hybrid work model. The ideal candidate has over 5 years of experience with BigData, Hadoop, and cloud technologies. Responsibilities include managing project deliveries and collaborating with developers to design efficient data solutions. This position offers competitive compensation and opportunities for professional growth.

Qualifications

• 5+ years implementing data applications or data platforms.
• 5+ years support experience of Big Data technologies in Hadoop ecosystem.
• Experience on implementing solutions with workflow orchestration tools.

Responsibilities

• Manage and ensure delivery of project tasks as required.
• Collaborates with developers to identify business requirements.
• Provide expert level IT technical lead services.

Skills

BigData/Hadoop

Python

Java

Spark

ETL/ELT tools

Airflow

CI/CD pipelines

Agile methodology

Education

Masters or Bachelor's degree in computer science or related discipline

Tools

AWS Glue ETL

Cloudera

Terraform

Gitlab

Jira

Job description

About the job Cloud Data Platform Engineer

Cloud Data Platform Engineer needs 5+ years implementing data applications or data platforms with BigData/Hadoop, Python/Java/Spark full stack, etc.

5+ year support experience of Big Cloud Data Platform Engineer requires:

• 3 days a week, hybrid
• Locations: Iselin, NJ, Charlotte, NC, NY, NY
• Infrastructure 30-40%. (Moving Data in & Out of cloud)
• Snowflake exp is preferred
• Masters or Bachelor's degree in computer science or related discipline
• Manage and ensure delivery of project tasks as required
• Excellent documentation skills to create and manage design, implementation and automation related documentation
• Implementation experience for Hadoop distribution platforms like Cloudera or AWS EMR.
• Extensive experience in designing, engineering and managing data lake ingestion, validation, transformation and consumption services leveraging cloud data tools like Hive, Spark, EMR, Glue ETL and Catalog, Snowflake etc.
• Experience on implementing solutions with Workflow orchestration/scheduling tools such as Airflow, Autosys etc.
• Experience building CI/CD pipelines for infrastructure implementations with Terraform, Gitlab etc.
• Must be ability to multi-task and prioritize tasks for self
• Must experience working in agile environment with multiple priorities

Preferred Skills:

• 5+ years implementing data applications or data platforms with BigData/Hadoop, Python/Java/Spark full stack, etc.
• 5+ year support experience of Big Data technologies in Hadoop ecosystem Hive, HDFS, MapReduce, Spark, Yarn, Kafka, Pig, HBase, Sqoop, Elastic Search, Kerberos.
• 5+ years experience with ETL/ELT tools such as AWS Glue ETL, Talend, Datastage etc.
• 5+ years experience with Orchestration tools such as Airflow, Autosys etc.
• Experience with Jira and Agile methodology is a plus;
• Experience with understanding DevOps or CI/CD process w.r.t to infrastructure management activities
• Design, develop and deliver cloud datastore solutions and develop automation pipelines to migrate data sets from On-prem to Cloud platforms. Practice Infrastructure as code to develop automation routines and integration flows to manage state of the datastore platform systems
• Provision secure from start datastores and enable them with required security controls including encryption, masking, certificate/keys rotation etc.
• Collaborates with developers, analysts, various system administrators to identify business requirements in designing efficient datastore solutions and interfaces.
• Identifies and documents all system constraints, implications, and consequences of various proposed system changes.
• Reviews technical documentation to guide system users and to assist with the ongoing operation, maintenance, and development of the system. Evaluates the efficiency and effectiveness of application operations and troubleshooting problems.
• Provide expert level IT technical lead services, including the direction, evaluation, selection, configuration, implementation, and integration of new and existing technologies and tools in a cloud platform.