Cloud Data Engineer - Spark, PySpark & DataBricks

Highbrow LLC; Town of Texas (WI); Hybrid; USD 120,000 - 160,000; Full time

Job summary

Highbrow LLC is seeking a Cloud Engineer to work with Austin-based leadership to design and implement data extraction and transformation pipelines using Spark, PySpark, and DataBricks.

The role involves writing complex SQL across Teradata, Hive, and Spark, building DevOps pipelines, and contributing to agile ceremonies. 3 days/week on-site at the Austin office with collaboration across teams.

Qualifications

• Strong development experience in Spark, PySpark, Shell scripting, Teradata, and DataBricks.
• Experience with Ab Initio is required.
• Strong experience in writing complex and effective SQLs ( Teradata, Hive, Spark ) and Stored Procedures.
• Proficiency in Shell Scripting for automation, system administration, and data processing tasks.
• Excellent work experience on DataBricks as data warehouse/Data Lake implementations.
• Experience in Agile and DevOps tools (Git, Jenkins, Artifactory), Unix/Linux Shell scripting (KSH) and basic Unix server administration.
• Hands-on experience with Teradata data modeling, SQL development, performance tuning, and large-scale data management.
• Experience in Jira and Confluence.

Responsibilities

• Collaborate with leadership to understand requirements and design solutions.
• Write product-grade code for data extraction, transformation, and loading using Spark/PySpark.
• Model data and write performant queries using Spark SQL, Teradata SQL and Hive SQL.
• Implement DevOps pipelines to deploy code artifacts to AWS/DataBricks.
• Orchestrate DataBricks jobs and monitor production data pipelines.
• Participate in agile ceremonies like sprint planning, grooming, daily scrums, and demos.

Skills

Spark/PySpark

Shell scripting

Teradata

DataBricks

Ab Initio

SQL (Teradata/Hive/Spark)

Unix/Linux Shell scripting (KSH)

DevOps practices

Git/Jenkins/Artifactory

Databricks Delta Lake

Tools

Jira

Confluence

Git

Jenkins

Artifactory

CA7 Scheduler

AWS (S3, EC2, Lambda, etc.)

DataBricks Platform

Unix Administration

Job description

Highbrow LLC is seeking a Cloud Engineer to work with Austin-based leadership to design and implement data extraction and transformation pipelines using Spark, PySpark, and DataBricks.

The role involves writing complex SQL across Teradata, Hive, and Spark, building DevOps pipelines, and contributing to agile ceremonies. 3 days/week on-site at the Austin office with collaboration across teams.