Senior Data Engineer - Remote: Cloud Data Pipelines & ML

Taleo; Eden Prairie (MN); Remote; USD 92,000 - 164,000; Full time

Benefits offered by this job

Telecommute option

Healthcare benefits

Job summary

Optum is seeking a Senior Data Engineer on the Optum Technology team to design scalable, cloud-native data pipelines using PySpark, Delta Lake, and Azure Data Factory. You will collaborate with Data Science teams to operationalize ML models and ensure data quality, security, and governance across healthcare data.

You can telecommute from anywhere in the U.S. This role requires 4+ years in data engineering with Databricks and Azure, plus CI/CD experience.

Qualifications

• 4+ years of Python and PySpark experience in data engineering.
• 4+ years building scalable ETL/ELT data pipelines with Databricks.
• 4+ years in Agile/Scrum project environments.
• 4+ years hands-on with Azure Databricks and Delta Lake.
• 4+ years CI/CD pipelines using GitHub Actions or Azure DevOps.
• 3+ years Azure cloud services including ADF or ADLS.
• 2+ years with unit testing, data validation, and pipeline testing.

Responsibilities

• Design and develop scalable ETL/ELT data pipelines using Databricks, Apache Spark, and Python on Azure cloud platforms.
• Build cloud-native data solutions leveraging Azure Data Lake Storage, Azure Data Factory, and Delta Lake architecture.
• Collaborate with Data Science and AI teams to operationalize ML models, feature stores, and real-time/streaming frameworks.
• Design, develop, and deploy AI-powered solutions to address complex business challenges with emphasis on responsible use of AI.
• Build reusable frameworks, engineering accelerators, and automation scripts to improve engineering efficiency and delivery quality.
• Ensure data quality, governance, security, and compliance across the complete data lifecycle.
• Optimize the performance, scalability, and cost-efficiency of batch and streaming data workflows, including Databricks Delta Live Tables.
• Participate in Agile development practices and provide production support/on-call coverage to ensure system stability.
• Leverage enterprise-approved AI tools to streamline software engineering workflows and drive continuous improvement.

Skills

Python

PySpark

Databricks

Azure

CI/CD

GitHub Actions

Azure DevOps

Education

Bachelor's degree in Computer Science or related field

Tools

Azure Data Factory

Azure Data Lake Storage

Delta Lake

Databricks

GitHub Actions

Azure DevOps

Job description

Optum is seeking a Senior Data Engineer on the Optum Technology team to design scalable, cloud-native data pipelines using PySpark, Delta Lake, and Azure Data Factory. You will collaborate with Data Science teams to operationalize ML models and ensure data quality, security, and governance across healthcare data.

You can telecommute from anywhere in the U.S. This role requires 4+ years in data engineering with Databricks and Azure, plus CI/CD experience.