Senior AWS Data Engineer: Databricks & Spark Lead(Hybrid)

VY Systems; St. Louis (MO); Hybrid; USD 140,000 - 180,000; Full time

Job summary

VY Systems is seeking a Lead AWS Data Engineer with Databricks expertise to guide enterprise data pipelines in a hybrid role based in St. Louis, MO. You will architect scalable cloud data platforms, drive modernization in the utility domain, and mentor engineers across teams.

Ideal candidates bring 6+ years in data engineering, deep AWS data service experience, Databricks and Spark proficiency, and a strong background in data governance and modeling for large datasets.

Qualifications

• Bachelor’s degree in Computer Science, Data Science, IT, Software Engineering, or related field.
• Minimum of 6+ years in data engineering, big data analytics, and cloud development.
• Hands-on expertise with AWS data services (S3, Glue, Redshift, Athena, EMR).
• Deep knowledge of Databricks, Spark, PySpark, and Delta Lake.
• Domain experience in the Utility sector is required.
• Advanced SQL, data modeling, and performance tuning for large data sets.
• Strong leadership, communication, and mentoring abilities.
• Ability to operate in a hybrid work model in St. Louis, MO.

Responsibilities

• Architect, design, and maintain enterprise‑scale data pipelines and data lakes using AWS and Databricks.
• Develop scalable ETL/ELT data pipelines across cloud data stores.
• Model complex data schemas for utility domain analytics.
• Monitor and optimize Databricks clusters and AWS storage (S3, Redshift, Glue, RDS).
• Ensure data governance, quality, and security across pipelines.
• Lead code reviews, mentor junior engineers, and promote best practices.
• Troubleshoot pipeline failures and performance bottlenecks.
• Collaborate with product, analytics, and infra teams in a hybrid model.
• Create technical docs, architecture diagrams, and runbooks.

Skills

AWS data services

Databricks

Apache Spark

PySpark

Delta Lake

SQL & data modeling

leadership & mentoring

Education

Bachelor's degree in CS/IT/related

Tools

Databricks

Apache Spark

PySpark

AWS (S3, Redshift, Glue, EMR)

Job description

VY Systems is seeking a Lead AWS Data Engineer with Databricks expertise to guide enterprise data pipelines in a hybrid role based in St. Louis, MO. You will architect scalable cloud data platforms, drive modernization in the utility domain, and mentor engineers across teams.

Ideal candidates bring 6+ years in data engineering, deep AWS data service experience, Databricks and Spark proficiency, and a strong background in data governance and modeling for large datasets.