Data Engineer – Real-Time Pipelines & Snowflake Mastery

The Digital Group Infotech; United States; Remote; USD 120,000 - 190,000; Full time

Job summary

The Digital Group Infotech is seeking a Data Engineer to design and optimize scalable batch and real-time data pipelines using Python, SQL, PySpark, and dbt. You will build ETL/ELT workflows and Snowflake-based data solutions for analytics.

Additionally you will develop ML feature pipelines, work with OpenAI/Azure OpenAI, LangChain tools, and implement data governance, quality checks, and RBAC within cloud-native platforms on AWS and Azure.

Qualifications

• Proficient in building scalable batch and real-time data pipelines.
• Experience with ETL/ELT workflows for diverse data types.
• Knowledge of data ingestion from APIs, databases, cloud storage, streaming platforms.

Responsibilities

• Design, develop, and maintain scalable batch and real-time data pipelines.
• Build and optimize ETL/ELT workflows for structured, semi-structured, and unstructured data.
• Develop reusable data ingestion frameworks for APIs, databases, cloud storage, streaming platforms, and third-party systems.
• Design and optimize Snowflake Data Cloud solutions including databases, schemas, tables, views, tasks, streams, Snowpark, and stored procedures.
• Develop dbt models, macros, snapshots, seeds, and tests following software engineering practices.
• Implement data quality checks, lineage, and automated tests using dbt.
• Design dimensional models (Star/Snowflake schemas) for analytics and reporting.
• Optimize SQL queries and data-processing workloads for performance.
• Implement data governance, security, masking, and RBAC.
• Develop Python libraries and utilities for data processing and engineering.
• AI/ML workflow: build ML features, pipelines, and deploy ML models with MLOps practices.

Skills

Python

SQL

PySpark

dbt

Data pipelines

ETL/ELT

Data modeling

Tools

Snowflake

Databricks

Apache Spark

Apache Airflow

Azure Data Factory

AWS Glue

LangChain

LlamaIndex

Job description

The Digital Group Infotech is seeking a Data Engineer to design and optimize scalable batch and real-time data pipelines using Python, SQL, PySpark, and dbt. You will build ETL/ELT workflows and Snowflake-based data solutions for analytics.

Additionally you will develop ML feature pipelines, work with OpenAI/Azure OpenAI, LangChain tools, and implement data governance, quality checks, and RBAC within cloud-native platforms on AWS and Azure.