Lead AWS Data Engineer with Databricks | St. Louis, MO

VY Systems; St. Louis (MO); Ibrido; USD 140.000 - 190.000; Tempo pieno

Vantaggi offerti da questo lavoro

Hybrid work schedule

Competitive compensation

Descrizione del lavoro

VY Systems seeks a Lead AWS Data Engineer with Databricks in St. Louis, MO (Hybrid). You will architect and optimize scalable data pipelines, data warehousing, and analytics platforms for the utility sector. Bring expert AWS, Databricks, Spark, and Delta Lake skills to lead complex ETL/ELT projects.

You will mentor engineers, drive governance, and collaborate with business analysts and software teams to deliver high-impact solutions in a hybrid work setup.

Competenze

• Bachelor's degree in CS, IT, or related field.
• 7+ years of software or data engineering experience.
• Strong AWS data services experience (S3, Glue, EMR, Redshift).
• Deep Databricks, Spark, PySpark, Delta Lake expertise.
• Proven ETL/ELT pipeline design and governance experience.

Mansioni

• Architect and build scalable ETL/ELT pipelines on AWS for utility domain analytics.
• Lead Databricks-based data processing and analytics development.
• Design data models and schemas (star/snowflake) for data warehousing.
• Mentor junior engineers and conduct code reviews.

Conoscenze

Leadership

Mentoring

Problem solving

Cross-functional communication

Stakeholder management

Formazione

Bachelor's degree in Computer Science or related field

Strumenti

AWS Cloud

Databricks

Apache Spark

Delta Lake

PySpark

SQL

ETL/ELT

Terraform

CI/CD (GitHub Actions/GitLab)

Airflow

Descrizione del lavoro

Position Summary

We are seeking an exceptionally skilled and experienced Lead AWS Data Engineer with Databricks to join our technology team on a hybrid contract basis in St. Louis, MO. In this critical senior data engineering and technical leadership role, you will be responsible for designing, building, and optimizing scalable data pipelines, cloud data warehouses, and distributed analytics platforms within the utility domain. The ideal candidate will bring advanced professional expertise in AWS cloud services, Apache Spark, Databricks, and complex ETL/ELT architecture. If you possess a distinguished background in big data engineering and energy/utility data systems, we invite you to apply.

Detailed Job Description

As a Lead AWS Data Engineer with Databricks in St. Louis, MO (Hybrid), you will serve as the principal technical authority responsible for architecting high-performance data processing pipelines and analytics infrastructure. Your daily responsibilities will encompass developing robust ETL/ELT workflows on AWS, optimizing large-scale data processing jobs using Databricks and PySpark, ensuring data governance, and collaborating closely with utility domain business analysts and software engineers. You will drive data architecture modernization, mentor junior data engineers, and oversee production deployments.

Key Responsibilities

• Architect, design, build, and maintain scalable ETL/ELT data pipelines and streaming data ingestion workflows on AWS.
• Lead big data processing and analytics development utilizing Databricks, Apache Spark, and Delta Lake frameworks.
• Design and implement optimized data models, star/snowflake schemas, and data mart structures tailored for utility domain analytics.
• Provision, configure, and manage cloud infrastructure components on Amazon Web Services (AWS) including S3, Glue, EMR, and Redshift.
• Write highly optimized, complex SQL queries, stored procedures, and PySpark transformation scripts.
• Collaborate proactively with utility domain experts, product managers, and enterprise architects to gather data requirements.
• Implement data quality validations, automated testing frameworks, and data governance protocols across cloud data stores.
• Monitor, troubleshoot, and optimize data pipeline performance, reducing job execution latency and compute costs.
• Mentor junior data engineers, conduct code reviews, and establish engineering best practices.
• Author comprehensive technical documentation, data architecture blueprints, and pipeline operational runbooks.

Required Qualifications & Skills

• Bachelor’s degree in Computer Science, Data Engineering, Information Technology, Software Engineering, or a related discipline.
• Minimum of 7+ years of professional software engineering and data engineering experience.
• Strong hands‑on professional expertise with AWS cloud services (S3, Glue, EMR, Redshift, Lambda, IAM).
• Deep technical mastery in Databricks, Apache Spark, PySpark, and Delta Lake architectures.
• Proven experience designing, building, and monitoring robust ETL/ELT data pipelines in enterprise environments.
• Strong domain knowledge or prior project experience within the Utility or Energy sector.
• Outstanding analytical problem‑solving, performance tuning, and cross‑functional communication capabilities.
• Mandatory residency/work authorization: Ability to operate on a hybrid schedule in St. Louis, MO.

Nice-to-Have Skills

• Professional certifications such as AWS Certified Data Analytics – Specialty or Databricks Certified Professional Data Engineer.
• Experience with infrastructure-as-code tools (Terraform) and CI/CD deployment pipelines (GitHub Actions, GitLab CI).
• Familiarity with modern workflow orchestration tools such as Apache Airflow or Prefect.
• Immediate availability for hybrid contract mobilization in St. Louis.

Application Information

• Salary/Rate: Competitive market rate
• Deadline: Immediate
• Notice Period: Immediate to 15 Days
• Contract Duration: Contract (Hybrid – St. Louis, MO)

Recruitment Pro Tip

When applying for this Lead AWS Data Engineer position, ensure your resume explicitly highlights your Databricks/PySpark portfolio, your AWS cloud data engineering experience, and any specific utility domain projects you have delivered.