Pyspark Architect

Avance Consulting; Charlotte (NC); On-site; USD 100,000 - 130,000; Full time

Job summary

Avance Consulting is looking for a technology consultant in Charlotte, NC, to engage in enterprise architecture, focusing on the Cloudera Hadoop ecosystem. Responsibilities include analyzing system requirements, developing integration strategies, and collaborating with development teams. Ideal candidates should have strong experience in technology consulting, with a robust understanding of architectural frameworks and ETL/ELT processes. Proficiency with tools such as Hadoop, Python, and Kafka is required, making this a great opportunity for those passionate about scalable and secure architectures.

Qualifications

• Strong experience in technology consulting and architectural frameworks.
• Deep knowledge of Cloudera Hadoop ecosystem including Hive, Scala, and SPARK.
• Experience with ETL/ELT frameworks and large data sets.

Responsibilities

• Gather and analyze system requirements for performance and scalability.
• Contribute to tech framework development and architectural definitions.
• Collaborate with development teams to resolve technical issues.

Skills

Technology consulting

Enterprise and solutions architecture

Cloudera Hadoop ecosystem

ETL/ELT framework design

Parallel data processing

REST APIs

Tools

Hadoop

Hive

Python

NoSQL databases

Kafka

Databricks

Job description

Responsibilities

• Participate in customer discussions to gather and analyze system requirements for performance, security and scalability.
• Provide suggestions on solution architecture, contribute to tech framework development, support enterprise architecture definition, and drive design creation
• Identify suitable technologies and tools available in the market, and perform a comparative analysis in line with client's business requirement, thereby identifying optimal architectural solution options
• Participate in developing integration strategies based on validation of proof of concept of proposed architectural solutions
• Facilitate a smooth transition by ensuring knowledge transfer and adherence to quality standards, while contributing to the build and deployment automation strategy
• Collaborate with development teams to resolve complex technical issues using solution deployment and roll out methods, during build, deployment and support phases
• Analyze system performance basis architectural solutions implemented and support in identifying potential risks in the system landscape
• Create artifacts of architectural frameworks and technical solutions, thereby enriching team's knowledge repository

Your contribution to the team

• The ability to analyse, design, and develop scalable and secure architectures.
• A knack for assessing multiple technologies, tools and frameworks to recommend optimal solutions.
• Understanding of integration methodologies, proof-of-concept validation and deployment strategies.
• The ability to document architectural frameworks and solutions.
• Proactive communication with internal teams and clients, ensuring alignment with business objectives.
• A knack to keep with up technical advancements to ensure relevant and effective recommendations.
• The ability to maintain standards and ensure smooth transition through knowledge transfer

Required Skill and Experience

• Strong experience in technology consulting, enterprise and solutions architecture and architectural frameworks
• Strong experience in Cloudera Hadoop ecosystem, i.e. Hadoop, Hive, Scala, SPARK, Sqoop, Flume, Kafka, Python, NoSQL database technologies
• Good Understanding and knowledge of Hadoop architecture and various components such as high availability architecture, experience in workload management, scalability and distributed platform architecture
• Experience with design and implementation of ETL/ELT framework for complex warehouses/marts. Knowledge of large data sets and experience with performance tuning and troubleshooting
• Background in all aspects of software engineering with strong skills in parallel data processing, data flows, REST APIs, JSON, XML, and micro service architecture

Preferred Skill and Experience

• Experience in working in any hyperscalers, AWS, Azure, GCP, Databricks and Snowflake
• Experience in data virtualization/data federationfor creating data product (e.g. in Starburst) in heterogenous environment (Hadoop, RDBMS, NoSQL etc.)
• Hands-on development mentality, with a willingness to troubleshoot and solve complex problems