Data Engineer: Spark, PySpark & Hive on Cloudera

Prism IT Global; Irving (TX); On-site; USD 120,000 - 150,000; Full time

Job summary

Prism IT Global is seeking a Data Engineer to design, build, and optimize scalable data pipelines using Apache Spark, PySpark, and Hive within Cloudera/AWS environments.

You will develop ETL/ELT processes, manage data warehousing structures, and ensure high performance through tuning, partitioning, and efficient storage formats. Collaboration with data scientists and analytics teams is essential.

Qualifications

• Proficiency with Spark architecture, drivers, executors and DAGs.
• Strong Python programming with PySpark for data transformations.
• Experience with HiveQL/ANSI SQL and partitioning strategies.
• Knowledge of optimized big data formats (Parquet/ORC/Avro).
• Solid understanding of dimensional modeling and data warehouse concepts.

Responsibilities

• Design, build and maintain scalable ETL/ELT pipelines using PySpark and Spark SQL/Hive.
• Optimize data layout, partitioning and indexing for performance.
• Tune Spark jobs and manage memory to reduce skew and bottlenecks.
• Ingest high-volume structured and unstructured data into the data ecosystem.
• Implement automated workflows with Airflow or similar schedulers.
• Collaborate with data scientists and analysts to translate requirements into data solutions.

Skills

Apache Spark

PySpark

HiveQL

Python

Data modeling

ETL

Tools

Parquet/ORC/Avro

Spark SQL

Hive

Airflow

AWS EMR

Databricks

Job description

Prism IT Global is seeking a Data Engineer to design, build, and optimize scalable data pipelines using Apache Spark, PySpark, and Hive within Cloudera/AWS environments.

You will develop ETL/ELT processes, manage data warehousing structures, and ensure high performance through tuning, partitioning, and efficient storage formats. Collaboration with data scientists and analytics teams is essential.