Hadoop and PySpark Developer

Avance Consulting; Plano (TX); On-site; USD 95,000 - 135,000; Full time

Job summary

Avance Consulting in Plano, TX seeks a Big Data Developer to contribute to requirements elicitation, design discussions, coding, and production issue resolution. The role emphasizes end-to-end SDLC, data warehouses, and Big Data stack experience.

The ideal candidate has hands-on Hadoop, Spark, PySpark, Python, SQL, and Unix shell scripting skills, with production Big Data experience in a fast-paced environment.

Qualifications

• Experience in Hadoop, Python, and Spark.
• Experience end-to-end implementation of data warehouse and data marts; PySpark development.
• Knowledge and hands-on experience in SQL, Unix shell scripting.
• Experience in working with a Big Data implementation in production environment.
• Experience with Big Data technologies like Hadoop, Hive, Spark, Python, etc.
• Experience in Python and Unix shell scripting.

Responsibilities

• Contribute to requirements elicitation by documenting business requirements.
• Facilitate software design discussions and document design decisions for the technical team.
• Participate in coding and integrate features into existing applications while maintaining stability.
• Conduct code reviews and maintain code repositories.
• Implement test strategies, analyze results, and coordinate bug fixes.
• Develop user training programs, documentation, and support frameworks.
• Resolve production issues and recommend preventive measures to improve reliability.
• Maintain detailed records of code, testing techniques, and support activities.

Skills

End-to-end SDLC

Cross-functional collaboration

Effective communication

Testing and troubleshooting

Tools

Hadoop

Hive

Spark

PySpark

Python

SQL

Unix shell

Job description

Job Type: Permanent

Job Description

• Contribute to the requirements elicitation process by documenting assigned parts of business requirements, in line with guidance provided
• Facilitate software application design discussions, and document design decisions to guide the technical team towards building software solutions
• Participate in coding and integrate new features or updates into existing applications, with a focus on maintaining system stability
• Conduct code reviews, do changes to the codebase and maintain code repositories
• Implement test strategies, analyse results, and coordinate bug fixes to uphold the software quality standards
• Develop user training programs, documentation, and support frameworks to ensure a smooth transition to new software applications
• Actively participate in resolving production issues and recommend preventive strategies to enhance system reliability
• Maintain detailed records of code, testing techniques, and support activities to enrich the knowledge base and assist other similar projects

Your contribution to the team:

• A collaborative spirit and excellent communication skills.
• The ability to handle end to end SDLC phases from requirement gathering to implementation.
• A knack for translating complex requirements into actionable development tasks.
• A proactive approach to testing, troubleshooting, and refining our applications.
• The ability to work with cross-functional teams and do software integration.

Required Skill and Experience

• Experience in Hadoop, Python, and Spark .
• Good experience in end-to-end implementation of data warehouse and data marts and Pyspark Development
• Knowledge and hands-on experience in SQL, Unix shell scripting.
• Experience in working with a Big Data implementation in production environment.
• Experience with Big Data technologies like Hadoop, Hive, Spark, python, etc.
• Experience in python and Unix shell scripting.

Preferred Skill and Experience

• Understanding of Agile methodologies and technologies.
• Able to understand and explore the constantly evolving tools within Hadoop ecosystem and apply them appropriately to the relevant problems at hand.