Lead Cloud Data Engineer — AWS & Databricks Architect

Univedge Consulting LLC; St. Louis (MO); On-site; USD 120,000 - 180,000; Full time

Job summary

Univedge Consulting LLC is seeking a senior data engineer to design and implement scalable, secure AWS and Databricks data platforms. You will build ETL/ELT pipelines with PySpark, manage Delta Lake governance, and optimize queries and pipelines for performance and cost.

Experience with Terraform for infra-as-code, and CI/CD with Jenkins and GitHub Actions is required. The role involves integrating diverse data sources, ensuring data quality, lineage, security, and compliance while driving

Qualifications

• Bachelor's degree in Computer Science, Information Technology, or related field.
• AWS Solutions Architect - Professional certification.
• Databricks Data Engineer Professional or Apache Spark certification.

Responsibilities

• Design and architect scalable, secure AWS and Databricks solutions using AWS networking and PaaS services.
• 2,

Skills

Advanced AWS networking

Real-time data streaming

SQL optimization

Distributed data processing

Python

PySpark

Data governance

Education

Bachelor's degree in Computer Science or related field

AWS Certified Solutions Architect - Professional

Databricks Certified Data Engineer Professional or Apache Spark certification

Tools

Databricks

Delta Lake

Unity Catalog

Delta Live Tables

Terraform

Jenkins

GitHub Actions

AWS Kinesis

AWS Redshift

AWS Elastic Cache

AWS Cognito

EKS

Job description

Univedge Consulting LLC is seeking a senior data engineer to design and implement scalable, secure AWS and Databricks data platforms. You will build ETL/ELT pipelines with PySpark, manage Delta Lake governance, and optimize queries and pipelines for performance and cost.

Experience with Terraform for infra-as-code, and CI/CD with Jenkins and GitHub Actions is required. The role involves integrating diverse data sources, ensuring data quality, lineage, security, and compliance while driving