Senior Data Engineer - Remote

Duobridge Talent Solution; United States; Remote; USD 130,000 - 170,000; Full time

Job summary

Duobridge Talent Solution is seeking a Senior Data Engineer with strong Python and data engineering experience to build scalable data platforms and pipelines. The role focuses on implementing data workflows using Azure, Databricks, and Snowflake across ingestion, processing, and analytics layers.

You will design robust data architectures, integrate REST/GraphQL APIs, and collaborate with AI engineers, product teams, and stakeholders to deliver reliable data solutions at scale.

Qualifications

• 5+ years of Data Engineering experience.
• Strong Python and Advanced SQL skills.
• Hands-on experience with Databricks, PySpark, Delta Lake, and Snowflake.
• Strong knowledge of Azure Data Factory, ADLS Gen2, and Azure Synapse.
• Experience with REST/GraphQL APIs, OAuth2/JWT, and API integrations.
• Knowledge of data architecture, CDC, Medallion Architecture, and dimensional modeling.
• Experience with Git and CI/CD.

Responsibilities

• Design and develop scalable ETL/ELT data pipelines.
• Build data ingestion frameworks using Python, APIs, and distributed processing.
• Develop Databricks pipelines using PySpark and Delta Lake.
• Work with Snowflake, Azure Data Factory, ADLS Gen2, and Azure Synapse.
• Design Medallion Architecture, CDC, data models, and data quality frameworks.
• Develop REST/GraphQL API integrations with authentication, retry, and error handling.
• Optimize pipeline performance, reliability, and cloud costs.
• Support batch, real-time, and event-driven data processing.
• Collaborate with AI engineers, architects, product teams, and business stakeholders.

Skills

Python

Advanced SQL

Databricks

PySpark

Delta Lake

Snowflake

Azure Data Factory

ADLS Gen2

Azure Synapse

REST/GraphQL APIs

OAuth2/JWT

API Integrations

Git

CI/CD

Data Modeling

Medallion Architecture

CDC

Job description

Job Summary:

We are looking for a Senior Data Engineer with strong Python and data engineering experience to build scalable data platforms and pipelines using Azure, Databricks, and Snowflake.

Key Responsibilities:

• Design and develop scalable ETL/ELT data pipelines.
• Build data ingestion frameworks using Python, APIs, and distributed processing.
• Develop Databricks pipelines using PySpark and Delta Lake.
• Work with Snowflake, Azure Data Factory, ADLS Gen2, and Azure Synapse.
• Design Medallion Architecture, CDC, data models, and data quality frameworks.
• Develop REST/GraphQL API integrations with authentication, retry, and error handling.
• Optimize pipeline performance, reliability, and cloud costs.
• Support batch, real-time, and event-driven data processing.
• Collaborate with AI engineers, architects, product teams, and business stakeholders.

Required Skills:

• 5+ years of Data Engineering experience.
• Strong Python and Advanced SQL skills.
• Hands-on experience with Databricks, PySpark, Delta Lake, and Snowflake.
• Strong knowledge of Azure Data Factory, ADLS Gen2, and Azure Synapse.
• Experience with REST/GraphQL APIs, OAuth2/JWT, and API integrations.
• Knowledge of data architecture, CDC, Medallion Architecture, and dimensional modeling.
• Experience with Git and CI/CD.

Good to Have:

Airflow, dbt, Kafka, Terraform, Azure OpenAI, and AI/ML data pipelines.