Senior Data Engineer (Databricks & Cloud Analytics)

System One; Salt Lake City (UT); Hybrid; USD 140,000 - 190,000; Full time

Job summary

System One in Salt Lake City, Utah, is seeking a Senior Data Engineer to design and implement enterprise data platforms and cloud analytics solutions. You will work with Databricks, Spark, Python, SQL, and Scala to build scalable pipelines in a hybrid on-site/remote environment.

This role involves managing Databricks components, optimizing queries, and designing ETL/ELT processes on Azure Data Factory, while collaborating in Agile teams and mentoring data engineering members.

Qualifications

• Bachelor's degree or equivalent professional experience.
• 6+ years designing and supporting enterprise data platforms or cloud data solutions.
• Hands-on with Databricks, Spark, Python, SQL, and Scala.
• Strong SQL development and performance tuning.
• ETL/ELT design on Azure Data Factory.
• Knowledge of Kafka or similar event streaming systems.
• Familiarity with cloud data storage (ADLS Gen2).
• Git version control proficiency; Linux scripting.
• Experience with Agile methodologies; mentoring.

Responsibilities

• Design, develop, and support enterprise data platforms and cloud analytics.
• Build scalable data pipelines using Databricks, Spark, Python, SQL, and Scala.
• Manage Databricks components: Workspaces, Delta Lake, Unity Catalog, Delta Live Tables, Databricks SQL, MLflow, Jobs.
• Optimize SQL queries and tune performance.
• Design ETL/ELT processes on Azure Data Factory.
• Work with Kafka and REST APIs for data integration.
• Use ADLS Gen2 and other cloud storage; collaborate in Linux environments.
• Use Git for version control; participate in Agile teams.
• Provide mentoring and leadership within data engineering teams.

Skills

Databricks

Apache Spark

Python

SQL

Scala

Agile

Mentoring

Git

Linux

Terraform

Azure DevOps

GitHub Actions

Delta Lake

Delta Live Tables

Kafka

Azure Data Factory

ADLS Gen2

Education

Bachelor's degree or equivalent experience

Tools

Databricks

Apache Spark

Python

SQL

Scala

Azure Data Factory

Kafka

Git

Linux

Terraform

Azure DevOps

GitHub Actions

Delta Lake

Delta Live Tables

ADLS Gen2

Job description

Job Title: Senior Data Engineer (Databricks & Cloud Analytics)

Location: Salt Lake City, Utah

Work Model: Hybrid -- onsite and remote

Responsibilities

• Design, develop, and support enterprise data platforms, cloud analytics solutions, or large-scale data engineering initiatives.
• Work hands-on with Databricks, Apache Spark (PySpark), Python, SQL, and Scala to build scalable data pipelines and analytics solutions.
• Manage and implement Databricks ecosystem components, including Workspaces, Delta Lake, Unity Catalog, Delta Live Tables, Databricks SQL, MLflow, and Databricks Jobs.
• Optimize SQL queries and improve query performance through tuning and best practices.
• Design and implement ETL/ELT processes utilizing cloud platforms such as Azure Data Factory.
• Work with event streaming technologies like Apache Kafka and integrate enterprise applications via REST APIs and web services.
• Utilize Azure Data Lake Storage (ADLS Gen2) and other cloud storage solutions.
• Collaborate within Linux environments, including shell scripting and command-line utilities.
• Use source control tools like Git for collaborative development.
• Participate in Agile teams practicing Scrum, Kanban, or SAFe methodologies.
• Provide technical mentoring and leadership within data engineering teams.

Requirements

• Bachelor's degree in Computer Science, Information Systems, Engineering, or related field or equivalent professional experience.
• At least 6 years of experience designing and supporting enterprise data platforms or cloud data solutions.
• Extensive hands-on experience with Databricks, Apache Spark, Python, SQL, and Scala.
• Proven SQL development skills, including query optimization and performance tuning.
• Experience designing ETL/ELT solutions with cloud platforms such as Azure Data Factory.
• Knowledge of Kafka or similar event streaming systems.
• Familiarity with cloud data storage (Azure Data Lake Storage or equivalents).
• Proficiency using Git for version control.
• Experience working in Linux environments and scripting.
• Preferred: Infrastructure as Code (Terraform), CI/CD pipelines (Azure DevOps, GitHub Actions).
• Strong analytical and troubleshooting skills.
• Prior experience working in Agile methodologies.
• Ability to prioritize multiple tasks independently and mentor team members.