Cloud Engineer (Client: -)

Highbrow LLC; Town of Texas (WI); On-site; USD 120,000 - 160,000; Full time

Job summary

Highbrow LLC is seeking a data engineering professional to design and implement data pipelines using Spark and PySpark. You will work with leadership to translate requirements into scalable ETL processes, write complex SQL across Teradata/Hive/Spark, and optimize data models for performance in a data warehouse/data lake environment.

Responsibilities include building robust data solutions, deploying DevOps pipelines, and orchestrating DataBricks jobs while collaborating in Agile ceremonies and

Qualifications

• Strong development experience in Spark, PySpark, and SQL across Teradata, Hive, and Spark SQL.
• Proficiency with Ab Initio and DataBricks platforms.
• Experience with shell scripting for automation and data processing tasks.
• Knowledge of DevOps tools (Git, Jenkins) and AWS cloud services.
• Hands-on data modeling, performance tuning, and large data management.

Responsibilities

• Design and implement data solutions based on business requirements.
• Write performant data extraction, transformation, and loading pipelines using Spark/PySpark.
• Develop and optimize SQL across Teradata, Hive, and Spark SQL.
• Operate DevOps pipelines and orchestrate DataBricks jobs.
• Collaborate with leadership in Agile ceremonies and deliver demos.

Skills

Spark

PySpark

Shell scripting

Teradata

SQL

Hive SQL

Stored Procedures

Tools

Ab Initio

DataBricks

Jira

Confluence

Git

Jenkins

AWS

Job description

Job Description

• – 3 days a week from Austin, TX office
• – Work with business and technical leadership to understand requirements.
• – Design to the requirements and document the designs.
• – Ability to write product-grade performant code for data extraction, transformations and loading using Spark, PySpark.
• – Do data modeling as needed for the requirements.
• – Write performant queries using Teradata SQL, Hive SQL and Spark SQL against Teradata and DataBricks Unity Catalog.
• – Implementing dev-ops pipelines to deploy code artifacts on to the designated platform/servers like AWS or DataBricks.
• – Implement DataBricks job orchestration using DataBricks.
• – Troubleshooting the issues, providing effective solutions and jobs monitoring in the production environment.
• – Participate in sprint planning sessions, refinement/story-grooming sessions, daily scrums, demos and retrospectives.

Qualifications

Required:

• – Strong development experience in Spark, PySpark, Shell scripting, Teradata, and DataBricks.
• – Experience with Ab Initio is required.
• – Strong experience in writing complex and effective SQLs (using Teradata SQL, Hive SQL and Spark SQL) and Stored Procedures.
• – Proficiency in Shell Scripting for automation, system administration, and data processing tasks.
• – Excellent work experience on DataBricks as data warehouse/Data Lake implementations.
• – Experience in Agile and working knowledge on DevOps tools (Git, Jenkins, Artifactory) Unix/Linux Shell scripting (KSH) and basic administration of Unix servers CA7 Enterprise Scheduler AWS (S3, EC2, SNS, SQS, Lambda, ECS, Glue, IAM, and CloudWatch) Databricks (Delta lake, Notebooks, Pipelines, cluster management, Azure/AWS integration).
• – Hands-on experience working with Teradata, including data modeling, SQL development, performance tuning, and large-scale data management.
• – Experience in Jira and Confluence Exercises considerable creativity, foresight, and judgment in conceiving, planning, and delivering initiatives.
• – Strong experience developing large-scale data processing solutions using Apache Spark and PySpark.

Desired:

• – Health care domain knowledge.