Cloud Engineer

Highbrow LLC; Town of Texas (WI); Hybrid; USD 120,000 - 160,000; Full time

Job summary

Highbrow LLC is seeking a Cloud Engineer to work with Austin-based leadership to design and implement data extraction and transformation pipelines using Spark, PySpark, and DataBricks.

The role involves writing complex SQL across Teradata, Hive, and Spark, building DevOps pipelines, and contributing to agile ceremonies. 3 days/week on-site at the Austin office with collaboration across teams.

Qualifications

• Strong development experience in Spark, PySpark, Shell scripting, Teradata, and DataBricks.
• Experience with Ab Initio is required.
• Strong experience in writing complex and effective SQLs ( Teradata, Hive, Spark ) and Stored Procedures.
• Proficiency in Shell Scripting for automation, system administration, and data processing tasks.
• Excellent work experience on DataBricks as data warehouse/Data Lake implementations.
• Experience in Agile and DevOps tools (Git, Jenkins, Artifactory), Unix/Linux Shell scripting (KSH) and basic Unix server administration.
• Hands-on experience with Teradata data modeling, SQL development, performance tuning, and large-scale data management.
• Experience in Jira and Confluence.

Responsibilities

• Collaborate with leadership to understand requirements and design solutions.
• Write product-grade code for data extraction, transformation, and loading using Spark/PySpark.
• Model data and write performant queries using Spark SQL, Teradata SQL and Hive SQL.
• Implement DevOps pipelines to deploy code artifacts to AWS/DataBricks.
• Orchestrate DataBricks jobs and monitor production data pipelines.
• Participate in agile ceremonies like sprint planning, grooming, daily scrums, and demos.

Skills

Spark/PySpark

Shell scripting

Teradata

DataBricks

Ab Initio

SQL (Teradata/Hive/Spark)

Unix/Linux Shell scripting (KSH)

DevOps practices

Git/Jenkins/Artifactory

Databricks Delta Lake

Tools

Jira

Confluence

Git

Jenkins

Artifactory

CA7 Scheduler

AWS (S3, EC2, Lambda, etc.)

DataBricks Platform

Unix Administration

Job description

Cloud Engineer (Client: ???)

[Location: Texas,]

Job Description

• 3 days a week from Austin, TX office
• Work with business and technical leadership to understand requirements.
• Design to the requirements and document the designs.
• Ability to write product-grade performant code for data extraction, transformations and loading using Spark, PySpark.
• Do data modeling as needed for the requirements.
• Write performant queries using Teradata SQL, Hive SQL and Spark SQL against Teradata and DataBricks Unity Catalog.
• Implementing dev-ops pipelines to deploy code artifacts on to the designated platform/servers like AWS or DataBricks.
• Implement DataBricks job orchestration using DataBricks.
• Troubleshooting the issues, providing effective solutions and jobs monitoring in the production environment.
• Participate in sprint planning sessions, refinement/story-grooming sessions, daily scrums, demos and retrospectives.

Qualifications

Required:

• Strong development experience in Spark, PySpark, Shell scripting, Teradata, and DataBricks.
• Experience with Ab Initio is required.
• Strong experience in writing complex and effective SQLs (using Teradata SQL, Hive SQL and Spark SQL) and Stored Procedures.
• Proficiency in Shell Scripting for automation, system administration, and data processing tasks.
• Excellent work experience on DataBricks as data warehouse/Data Lake implementations.
• Experience in Agile and working knowledge on DevOps tools (Git, Jenkins, Artifactory) Unix/Linux Shell scripting (KSH) and basic administration of Unix servers CA7 Enterprise Scheduler AWS (S3, EC2, SNS, SQS, Lambda, ECS, Glue, IAM, and CloudWatch) Databricks (Delta lake, Notebooks, Pipelines, cluster management, Azure/AWS integration).
• Hands-on experience working with Teradata, including data modeling, SQL development, performance tuning, and large-scale data management.
• Experience in Jira and Confluence Exercises considerable creativity, foresight, and judgment in conceiving, planning, and delivering initiatives.
• Strong experience developing large-scale data processing solutions using Apache Spark and PySpark.

Desired:

• Health care domain knowledge.