Spark & DataBricks Cloud Engineer (AWS, SQL, DevOps)

Highbrow LLC; Town of Texas (WI); On-site; USD 120,000 - 160,000; Full time

Job summary

Highbrow LLC is seeking a data engineering professional to design and implement data pipelines using Spark and PySpark. You will work with leadership to translate requirements into scalable ETL processes, write complex SQL across Teradata/Hive/Spark, and optimize data models for performance in a data warehouse/data lake environment.

Responsibilities include building robust data solutions, deploying DevOps pipelines, and orchestrating DataBricks jobs while collaborating in Agile ceremonies and

Qualifications

• Strong development experience in Spark, PySpark, and SQL across Teradata, Hive, and Spark SQL.
• Proficiency with Ab Initio and DataBricks platforms.
• Experience with shell scripting for automation and data processing tasks.
• Knowledge of DevOps tools (Git, Jenkins) and AWS cloud services.
• Hands-on data modeling, performance tuning, and large data management.

Responsibilities

• Design and implement data solutions based on business requirements.
• Write performant data extraction, transformation, and loading pipelines using Spark/PySpark.
• Develop and optimize SQL across Teradata, Hive, and Spark SQL.
• Operate DevOps pipelines and orchestrate DataBricks jobs.
• Collaborate with leadership in Agile ceremonies and deliver demos.

Skills

Spark

PySpark

Shell scripting

Teradata

SQL

Hive SQL

Stored Procedures

Tools

Ab Initio

DataBricks

Jira

Confluence

Git

Jenkins

AWS

Job description

Highbrow LLC is seeking a data engineering professional to design and implement data pipelines using Spark and PySpark. You will work with leadership to translate requirements into scalable ETL processes, write complex SQL across Teradata/Hive/Spark, and optimize data models for performance in a data warehouse/data lake environment.

Responsibilities include building robust data solutions, deploying DevOps pipelines, and orchestrating DataBricks jobs while collaborating in Agile ceremonies and