Lead AWS Data Engineer with Databricks | St. Louis, MO (Hybrid)

VY Systems; St. Louis (MO); Hybrid; USD 140,000 - 180,000; Full time

Job summary

VY Systems is seeking a Lead AWS Data Engineer with Databricks expertise to guide enterprise data pipelines in a hybrid role based in St. Louis, MO. You will architect scalable cloud data platforms, drive modernization in the utility domain, and mentor engineers across teams.

Ideal candidates bring 6+ years in data engineering, deep AWS data service experience, Databricks and Spark proficiency, and a strong background in data governance and modeling for large datasets.

Qualifications

• Bachelor’s degree in Computer Science, Data Science, IT, Software Engineering, or related field.
• Minimum of 6+ years in data engineering, big data analytics, and cloud development.
• Hands-on expertise with AWS data services (S3, Glue, Redshift, Athena, EMR).
• Deep knowledge of Databricks, Spark, PySpark, and Delta Lake.
• Domain experience in the Utility sector is required.
• Advanced SQL, data modeling, and performance tuning for large data sets.
• Strong leadership, communication, and mentoring abilities.
• Ability to operate in a hybrid work model in St. Louis, MO.

Responsibilities

• Architect, design, and maintain enterprise‑scale data pipelines and data lakes using AWS and Databricks.
• Develop scalable ETL/ELT data pipelines across cloud data stores.
• Model complex data schemas for utility domain analytics.
• Monitor and optimize Databricks clusters and AWS storage (S3, Redshift, Glue, RDS).
• Ensure data governance, quality, and security across pipelines.
• Lead code reviews, mentor junior engineers, and promote best practices.
• Troubleshoot pipeline failures and performance bottlenecks.
• Collaborate with product, analytics, and infra teams in a hybrid model.
• Create technical docs, architecture diagrams, and runbooks.

Skills

AWS data services

Databricks

Apache Spark

PySpark

Delta Lake

SQL & data modeling

leadership & mentoring

Education

Bachelor's degree in CS/IT/related

Tools

Databricks

Apache Spark

PySpark

AWS (S3, Redshift, Glue, EMR)

Job description

Position Summary

We are seeking an exceptionally skilled and experienced Lead AWS Data Engineer with Databricks expertise to join our client team on a hybrid contract basis in St. Louis, MO. In this critical data engineering leadership role, you will be responsible for architecting enterprise data pipelines, building scalable big data analytics platforms on AWS, and driving digital modernization within the utility domain. The ideal candidate will bring deep hands‑on expertise in AWS data services, Apache Spark, Databricks, and utility industry data architectures. If you possess a distinguished background in big data engineering and technical team leadership, we invite you to apply.

Detailed Job Description

As a Lead AWS Data Engineer with Databricks in St. Louis, MO (Hybrid), you will serve as the principal technical authority responsible for designing, building, and optimizing cloud-native data processing pipelines. Your daily responsibilities will encompass developing scalable ETL/ELT data pipelines using AWS and Databricks, implementing complex data models for the utility sector, managing big data cluster performance, and mentoring engineering team members. You will collaborate closely with data architects, business analysts, and IT stakeholders to ensure robust data governance and high data quality standards.

Key Responsibilities

• Architect, design, develop, and maintain enterprise‑scale data pipelines and data lakes using AWS cloud services and Databricks.
• Build high‑performance big data processing workflows using Apache Spark, PySpark, and distributed computing frameworks.
• Implement robust ETL/ELT data integration processes connecting disparate enterprise data sources with cloud data stores.
• Apply deep industry domain knowledge within the utility sector to design specialized data models and analytical solutions.
• Configure, monitor, and optimize Databricks clusters and AWS data storage repositories (S3, Redshift, Glue, RDS).
• Ensure strict adherence to data governance, data quality, data security, and compliance standards.
• Lead technical code reviews, mentor junior data engineers, and establish engineering best practices.
• Troubleshoot complex data pipeline failures, query performance bottlenecks, and infrastructure issues.
• Collaborate proactively with cross‑functional product, analytics, and infrastructure teams in a hybrid work model.
• Author comprehensive technical documentation, data architecture diagrams, and deployment runbooks.

Required Qualifications & Skills

• Bachelor’s degree in Computer Science, Data Science, Information Technology, Software Engineering, or a related discipline.
• Minimum of 6+ years of professional experience in data engineering, big data analytics, and cloud platform development.
• Strong hands‑on professional expertise with AWS data services (Amazon S3, AWS Glue, Redshift, Athena, EMR).
• Deep technical proficiency in Databricks, Apache Spark, PySpark, and Delta Lake architecture.
• Mandatory domain background: Practical experience and professional exposure within the Utility Domain.
• Advanced SQL querying, data modeling, and performance tuning capabilities for large‑scale datasets.
• Strong technical leadership, cross‑functional communication, and mentoring skills.
• Mandatory residency/work authorization: Ability to operate in a hybrid work model in St. Louis, MO.

Nice‑to‑Have Skills

• Professional certifications such as AWS Certified Data Analytics – Specialty or Databricks Certified Data Engineer Professional.
• Experience with CI/CD deployment pipelines, Git, and Terraform infrastructure as code.
• Familiarity with real‑time data streaming technologies such as Apache Kafka or AWS Kinesis.
• Immediate availability for hybrid contract mobilization in St. Louis, MO.

Application Information

• Salary/Rate: Competitive market rate
• Deadline: Immediate
• Notice Period: Immediate to 30 Days
• Contract Duration: Contract / Hybrid (St. Louis, MO)

Recruitment Pro Tip

When applying for this Lead AWS Data Engineer position, ensure your resume explicitly highlights your Databricks/Spark hands‑on experience, your AWS data pipeline portfolio, and your specific utility domain background.