Spark Engineer

Veriipro; Jacksonville (FL); On-site; USD 90,000 - 130,000; Full time

Job summary

Veriipro is seeking a data engineer skilled in Apache Spark to design and maintain scalable data processing solutions. The ideal candidate will possess strong experience with ETL pipelines, cloud platforms like AWS and Azure, and SQL-based data analysis.

The role requires collaboration with data teams to ensure high-quality data solutions and optimization for performance. Candidates should have a thorough understanding of distributed architectures and Spark internals.

Qualifications

• Strong hands-on experience in Apache Spark and distributed data processing.
• Deep understanding of Spark internals, architecture, and execution lifecycle.
• Expertise in SQL-based data transformation and analysis.
• Experience with ETL frameworks, data modeling, and cloud-based data platforms.
• Good understanding of distributed systems and scalable data architectures.

Responsibilities

• Design, develop, and maintain scalable batch and real-time data processing solutions using Apache Spark.
• Build and optimize ETL pipelines to support analytics and downstream applications.
• Develop high-performance data transformations using Spark SQL, DataFrames, Datasets, and RDDs.
• Perform Spark performance tuning, query optimization, and resource management for scalability and efficiency.
• Work with cloud-based storage and processing platforms in AWS/Azure environments.
• Collaborate with data engineers, architects, and analytics teams to deliver end-to-end data solutions.
• Ensure data quality, reliability, and adherence to engineering best practices.
• Support deployment, monitoring, and troubleshooting of Spark applications in distributed environments.

Skills

Apache Spark

ETL pipelines

Cloud platforms (AWS/Azure)

SQL

Data modeling

Linux/Unix

Tools

Snowflake

Amazon S3

Azure Data Lake Storage

Job description

Required Skills

• Strong hands‑on experience with Apache Spark 3.5.x or later.
• Expertise in Spark DataFrames, Datasets, RDDs, and Spark SQL.
• Strong understanding of Spark architecture, execution model, and distributed computing principles.
• Experience designing and building scalable ETL pipelines for batch and real‑time data processing.
• Proven experience in Spark job tuning and performance optimization.
• Hands‑on experience with cloud platforms such as AWS and/or Azure.
• Experience with cloud storage solutions including Amazon S3 and Azure Data Lake Storage (ADLS).
• Strong SQL and data modeling skills.
• Experience working in Linux/Unix environments.

Preferred Skills

• Experience with Snowflake for data warehousing and analytics.
• Exposure to streaming/real‑time data processing frameworks.
• Familiarity with large‑scale distributed data architectures.
• Knowledge of Scala programming language.

Roles & Responsibilities

• Design, develop, and maintain scalable batch and real‑time data processing solutions using Apache Spark.
• Build and optimize ETL pipelines to support analytics and downstream applications.
• Develop high‑performance data transformations using Spark SQL, DataFrames, Datasets, and RDDs.
• Perform Spark performance tuning, query optimization, and resource management for scalability and efficiency.
• Work with cloud‑based storage and processing platforms in AWS/Azure environments.
• Collaborate with data engineers, architects, and analytics teams to deliver end‑to‑end data solutions.
• Ensure data quality, reliability, and adherence to engineering best practices.
• Support deployment, monitoring, and troubleshooting of Spark applications in distributed environments.

Required Qualifications

• Strong hands‑on experience in Apache Spark and distributed data processing.
• Deep understanding of Spark internals, architecture, and execution lifecycle.
• Expertise in SQL‑based data transformation and analysis.
• Experience with ETL frameworks, data modeling, and cloud‑based data platforms.
• Good understanding of distributed systems and scalable data architectures.