Hadoop & PySpark Data Platform Engineer

Envision Technology Solutions; Addison (TX); On-site; USD 120,000 - 170,000; Full time

Job summary

Envision Technology Solutions is seeking a data architecture engineer to lead end-to-end data platform design in a hybrid on-site setting at Plano/Addison or Charlotte. The role focuses on building scalable data pipelines and real-time streaming solutions.

You will design and implement data ingestion, transformation, and processing pipelines using Hadoop, PySpark, and Python, integrate Kafka for event-driven architectures, and mentor junior engineers within an AGILE team.

Qualifications

• Strong experience with Hadoop ecosystem (HDFS, Hive, Spark).
• Proficiency in PySpark for distributed data processing.
• Advanced programming skills in Python.
• Hands-on experience with Apache Kafka for real-time streaming.
• Expertise in designing scalable, secure, and high-performance systems.
• Familiarity with microservices, API design, and cloud-native architectures.

Responsibilities

• Define end-to-end architecture for data platforms, streaming systems, and web applications.
• Ensure alignment with enterprise standards, security, and compliance requirements.
• Evaluate emerging technologies and recommend adoption strategies.
• Design and implement data ingestion, transformation, and processing pipelines using Hadoop, PySpark, and related tools.
• Optimize ETL workflows for large-scale datasets and real-time streaming.
• Integrate Apache Kafka for event-driven architectures and messaging.
• Work closely with product owners, business analysts, and DevOps teams.
• Mentor junior developers and data engineers.
• Participate in agile ceremonies, code reviews, and design discussions.

Skills

Hadoop ecosystem

PySpark

Python

Apache Kafka

Microservices

API design

Cloud-native architectures

Job description

Envision Technology Solutions is seeking a data architecture engineer to lead end-to-end data platform design in a hybrid on-site setting at Plano/Addison or Charlotte. The role focuses on building scalable data pipelines and real-time streaming solutions.

You will design and implement data ingestion, transformation, and processing pipelines using Hadoop, PySpark, and Python, integrate Kafka for event-driven architectures, and mentor junior engineers within an AGILE team.