Pyspark Architect

Avance Consulting; North Carolina; On-site; USD 100,000 - 130,000; Full time

Job summary

Avance Consulting is looking for a skilled technology consultant based in North Carolina. The role requires strong experience in the Cloudera Hadoop ecosystem, including Hive, Scala, and SPARK, as well as expertise in designing ETL/ELT frameworks for complex data warehouses.

The ideal candidate will have a background in software engineering and must be proficient in data virtualization and parallel data processing. This position offers the opportunity to solve complex problems in a dynamic environment.

Responsibilities

• Design and implement ETL/ELT frameworks for data warehouses.
• Work with large data sets; performance tuning and troubleshooting.
• Develop complex data products in heterogeneous environments.

Skills

Technology consulting

Cloudera Hadoop ecosystem

ETL/ELT framework

Data virtualization

Microservice architecture

Tools

AWS

Azure

GCP

Python

Scala

Hadoop

Job description

• Strong experience in technology consulting, enterprise and solutions architecture and architectural frameworks
• Strong experience in Cloudera Hadoop ecosystem, i.e. Hadoop, Hive, Scala, SPARK, Sqoop, Flume, Kafka, Python, NoSQL database technologies
• Good Understanding and knowledge of Hadoop architecture and various components such as high availability architecture, experience in workload management, scalability and distributed platform architecture
• Experience with design and implementation of ETL/ELT framework for complex warehouses/marts. Knowledge of large data sets and experience with performance tuning and troubleshooting
• Background in all aspects of software engineering with strong skills in parallel data processing, data flows, REST APIs, JSON, XML, and micro service architecture

Preferred Skill and Experience

• Experience in working in any hyperscalers, AWS, Azure, GCP, Databricks and Snowflake
• Experience in data virtualization/data federationfor creating data product (e.g. in Starburst) in heterogenous environment (Hadoop, RDBMS, NoSQL etc.)
• Hands-on development mentality, with a willingness to troubleshoot and solve complex problems