Hadoop / Bigdata Developer

Envision Technology Solutions; Charlotte (NC); On-site; USD 140,000 - 180,000; Full time

Job summary

Envision Technology Solutions seeks a senior data engineering/architect to define end-to-end architecture for data platforms and streaming systems. You will lead data pipelines using Hadoop, PySpark, and Kafka, while aligning with security and enterprise standards.

You will mentor engineers, collaborate with product owners and DevOps, and drive agile practices. Strong Python and distributed processing skills are required to thrive in a fast-paced environment.

Qualifications

• Minimum 10 years of hands-on data engineering and architecture experience.
• Deep expertise in Hadoop ecosystem with Spark, Hive, and HDFS.
• Proficiency in PySpark and Python for distributed processing and ETL.

Responsibilities

• Define end-to-end architecture for data platforms, streaming systems, and web applications.
• Lead data engineering initiatives and mentor junior developers and engineers.
• Collaborate with product owners, analysts, and DevOps; participate in agile ceremonies.
• Evaluate emerging technologies and recommend adoption strategies.

Skills

Hadoop ecosystem

PySpark

Python

Apache Kafka

Tools

HDFS

Hive

Spark

Apache Kafka

Job description

Experience: Minimum 10 years

Roles & Responsibilities

Must Have Technical/Functional Skills

Primary Skill: Hadoop ecosystem (HDFS, Hive, Spark), PySpark, Python, Apache Kafka

Experience: Minimum 10 years

Roles & Responsibilities

Architectural Leadership

• Define end-to-end architecture for data platforms, streaming systems, and web applications.
• Ensure alignment with enterprise standards, security, and compliance requirements.
• Evaluate emerging technologies and recommend adoption strategies.

Data Engineering

• Design and implement data ingestion, transformation, and processing pipelines using Hadoop, PySpark, and related tools.
• Optimize ETL workflows for large-scale datasets and real-time streaming.
• Integrate Apache Kafka for event-driven architectures and messaging.

Collaboration & Leadership

• Work closely with product owners, business analysts, and DevOps teams.
• Mentor junior developers and data engineers.
• Participate in agile ceremonies, code reviews, and design discussions.

Technical Expertise

Required Skills & Qualifications:

• Strong experience with Hadoop ecosystem (HDFS, Hive, Spark).
• Proficiency in PySpark for distributed data processing.
• Advanced programming skills in Python.
• Hands-on experience with Apache Kafka for real-time streaming.

Architectural Skills

• Expertise in designing scalable, secure, and high-performance systems.
• Familiarity with microservices, API design, and cloud-native architectures.

Additional Skills

• Knowledge of CI/CD pipelines, containerization (Docker/Kubernetes).
• Exposure to cloud platforms (AWS, Azure, GCP). Soft Skills:
• Strong analytical and problem-solving abilities.
• Excellent communication and leadership skills.
• Ability to thrive in a fast-paced, agile environment.