Spark Engineer: Real-Time ETL & Cloud Data Pipelines

Veriipro; Jacksonville (FL); On-site; USD 90,000 - 130,000; Full time

Job summary

Veriipro is seeking a data engineer skilled in Apache Spark to design and maintain scalable data processing solutions. The ideal candidate will possess strong experience with ETL pipelines, cloud platforms like AWS and Azure, and SQL-based data analysis.

The role requires collaboration with data teams to ensure high-quality data solutions and optimization for performance. Candidates should have a thorough understanding of distributed architectures and Spark internals.

Qualifications

• Strong hands-on experience in Apache Spark and distributed data processing.
• Deep understanding of Spark internals, architecture, and execution lifecycle.
• Expertise in SQL-based data transformation and analysis.
• Experience with ETL frameworks, data modeling, and cloud-based data platforms.
• Good understanding of distributed systems and scalable data architectures.

Responsibilities

• Design, develop, and maintain scalable batch and real-time data processing solutions using Apache Spark.
• Build and optimize ETL pipelines to support analytics and downstream applications.
• Develop high-performance data transformations using Spark SQL, DataFrames, Datasets, and RDDs.
• Perform Spark performance tuning, query optimization, and resource management for scalability and efficiency.
• Work with cloud-based storage and processing platforms in AWS/Azure environments.
• Collaborate with data engineers, architects, and analytics teams to deliver end-to-end data solutions.
• Ensure data quality, reliability, and adherence to engineering best practices.
• Support deployment, monitoring, and troubleshooting of Spark applications in distributed environments.

Skills

Apache Spark

ETL pipelines

Cloud platforms (AWS/Azure)

SQL

Data modeling

Linux/Unix

Tools

Snowflake

Amazon S3

Azure Data Lake Storage

Job description

Veriipro is seeking a data engineer skilled in Apache Spark to design and maintain scalable data processing solutions. The ideal candidate will possess strong experience with ETL pipelines, cloud platforms like AWS and Azure, and SQL-based data analysis.

The role requires collaboration with data teams to ensure high-quality data solutions and optimization for performance. Candidates should have a thorough understanding of distributed architectures and Spark internals.