Senior Data Engineer

Vytwo; Minneapolis (MN); On-site; USD 120,000 - 180,000; Full time

Benefits offered by this job

Flexible work from home options

Job summary

Vytwo in Minneapolis, MN seeks a Senior Data Engineer to design and maintain scalable data pipelines and optimize Spark workloads for analytics teams.

You will work with Databricks, Delta Lake, and SQL across batch and streaming data, collaborating with stakeholders to deliver reliable, analytics-ready datasets and ensure security and version control.

Qualifications

• 5+ years experience in Data Engineering or related role.
• Strong hands-on Apache Spark (PySpark or Scala).
• Proven Databricks experience.
• Strong SQL skills with relational and analytical databases.
• Experience building ETL/ELT pipelines at scale.
• Familiarity with Delta Lake architectures; Git-based version control.

Responsibilities

• Design, develop, and maintain scalable data pipelines using Apache Spark (PySpark/Scala).
• Build and optimize data workflows on Databricks, including Delta Lake and scheduled jobs.
• Ingest, transform, and curate large-scale structured and semi-structured datasets.
• Conduct performance tuning and cost optimization for Spark workloads and Databricks clusters.
• Implement data quality checks, monitoring, and error handling.
• Collaborate with analytics and business stakeholders to deliver analytics-ready data.
• Support batch processing and streaming data pipelines where applicable.

Skills

5+ years experience

Apache Spark

Databricks

SQL

Git

Delta Lake

PySpark/Scala

Job description

Vytwo is seeking a Senior Data Engineer to design and maintain scalable data pipelines and improve Databricks and Spark workloads so downstream analytics teams receive reliable, analytics-ready datasets. This role is based in Minneapolis, MN with onsite work.

Key Responsibilities

• Design, develop, and maintain scalable data pipelines using Apache Spark with PySpark and/or Scala.
• Build and optimize data workflows on Databricks, including Delta Lake, notebooks, and scheduled jobs.
• Ingest, transform, and curate large-scale structured and semi-structured datasets.
• Conduct performance tuning and cost optimization for Spark workloads and Databricks clusters.
• Implement data quality checks, monitoring, and error handling practices.
• Collaborate with analytics and business stakeholders to deliver well-modelled, analytics-ready data.
• Support batch processing and, where applicable, streaming data pipelines.
• Apply best practices for testing, documentation, security, and version control.

Required Qualifications

• 5+ years of experience in Data Engineering or a related role.
• Strong hands-on experience with Apache Spark (PySpark or Scala).
• Proven experience working in Databricks environments.
• Strong SQL skills, including experience with relational and analytical databases.
• Experience building and maintaining ETL/ELT pipelines at scale.
• Familiarity with modern data lake architectures; Delta Lake preferred.
• Experience using Git-based version control.
• Must be a US Citizen.

Technology Stack

• Apache Spark, PySpark, Scala
• Databricks, Delta Lake
• Git, SQL

Location and Eligibility

Minneapolis, MN (onsite). Must be currently located in Minnesota (MN), or willing to relocate. Must be a U.S. Citizen.

Benefits

• Flexible work from home options available.