Remote Senior Data Engineer: Elite Data Pipelines & AI

Red Hat, Inc.; Raleigh (NC); On-site; USD 158,000 - 180,000; Full time

Job summary

Red Hat, Inc. is seeking a Senior Data Engineer to architect and implement high-volume data pipelines between Snowflake and Databricks using PySpark and dbt for data transformation.

The role involves orchestrating Airflow workloads, securing S3 access, and building AI-driven retrieval and ML-based workflows. The successful candidate will manage Databricks IAM, secrets, and workspace governance, deploy ML models with MLflow, and contribute to security and compliance across the stack.

Qualifications

• Master's or Bachelor's degree in Computer Science or related field with relevant experience.
• 3+ years architecting high-volume data pipelines between Snowflake and Databricks using PySpark.

Responsibilities

• Orchestrate data pipelines and scheduling with Airflow for analytics workloads.
• Administer Databricks environments including IAM and secrets management.
• Design AI-driven workflows and enterprise data retrieval architectures.
• Implement MLOps with MLflow and monitor model performance post-production.
• Lead CI/CD pipelines and container deployments on OpenShift/Kubernetes.
• Conduct security assessments and threat modeling across the stack.

Skills

PySpark

Databricks

Snowflake

dbt

Airflow

OpenShift

Kubernetes

MLflow

Git

Tekton

Buildah

SAST

Threat Modeling

Education

Master's degree in Computer Science

Bachelor's degree in Computer Science

Tools

Terraform

Jenkins

SonarQube

Qualys

Splunk

Job description

Red Hat, Inc. is seeking a Senior Data Engineer to architect and implement high-volume data pipelines between Snowflake and Databricks using PySpark and dbt for data transformation.

The role involves orchestrating Airflow workloads, securing S3 access, and building AI-driven retrieval and ML-based workflows. The successful candidate will manage Databricks IAM, secrets, and workspace governance, deploy ML models with MLflow, and contribute to security and compliance across the stack.