Data Engineer

The Digital Group Infotech; United States; Remote; USD 120,000 - 190,000; Full time

Job summary

The Digital Group Infotech is seeking a Data Engineer to design and optimize scalable batch and real-time data pipelines using Python, SQL, PySpark, and dbt. You will build ETL/ELT workflows and Snowflake-based data solutions for analytics.

Additionally you will develop ML feature pipelines, work with OpenAI/Azure OpenAI, LangChain tools, and implement data governance, quality checks, and RBAC within cloud-native platforms on AWS and Azure.

Qualifications

• Proficient in building scalable batch and real-time data pipelines.
• Experience with ETL/ELT workflows for diverse data types.
• Knowledge of data ingestion from APIs, databases, cloud storage, streaming platforms.

Responsibilities

• Design, develop, and maintain scalable batch and real-time data pipelines.
• Build and optimize ETL/ELT workflows for structured, semi-structured, and unstructured data.
• Develop reusable data ingestion frameworks for APIs, databases, cloud storage, streaming platforms, and third-party systems.
• Design and optimize Snowflake Data Cloud solutions including databases, schemas, tables, views, tasks, streams, Snowpark, and stored procedures.
• Develop dbt models, macros, snapshots, seeds, and tests following software engineering practices.
• Implement data quality checks, lineage, and automated tests using dbt.
• Design dimensional models (Star/Snowflake schemas) for analytics and reporting.
• Optimize SQL queries and data-processing workloads for performance.
• Implement data governance, security, masking, and RBAC.
• Develop Python libraries and utilities for data processing and engineering.
• AI/ML workflow: build ML features, pipelines, and deploy ML models with MLOps practices.

Skills

Python

SQL

PySpark

dbt

Data pipelines

ETL/ELT

Data modeling

Tools

Snowflake

Databricks

Apache Spark

Apache Airflow

Azure Data Factory

AWS Glue

LangChain

LlamaIndex

Job description

Job Description :

Data Engineering :

• Design, develop, and maintain scalable batch and real-time data pipelines using Python, SQL, PySpark, and dbt.
• Build, optimize, and maintain ETL/ELT workflows for structured, semi-structured, and unstructured data.
• Develop scalable and reusable data ingestion frameworks to ingest data from APIs, databases, cloud storage, streaming platforms, and third-party systems.
• Design, develop, and optimize enterprise-grade Snowflake Data Cloud solutions, including databases, schemas, tables, views, tasks, streams, dynamic tables, Snowpark, and stored procedures.
• Develop reusable dbt models, macros, snapshots, seeds, and tests following software engineering best practices.
• Implement data quality checks, data lineage, documentation, and automated testing using dbt.
• Design and implement dimensional data models, including Star and Snowflake schemas, for analytics and reporting.
• Optimize SQL queries and data-processing workloads to support high-performance analytical applications.
• Implement data governance, security, data masking, and role-based access control (RBAC).
• Develop reusable Python libraries, utilities, and frameworks for data processing and engineering.

AI / Machine Learning:

• Design, develop, and deploy machine learning models for predictive analytics, NLP, recommendation systems, computer vision, and Generative AI applications.
• Build scalable feature engineering and data preprocessing pipelines for machine learning workloads.
• Train, evaluate, tune, and optimize machine learning models to improve performance and business outcomes.
• Design and develop Retrieval-Augmented Generation (RAG) pipelines using vector databases and Large Language Models (LLMs).
• Integrate OpenAI, Azure OpenAI, Anthropic, and AWS Bedrock APIs into enterprise applications and AI solutions.
• Build AI agents and workflow automation solutions using frameworks such as LangChain, LangGraph, and LlamaIndex.
• Deploy machine learning and Generative AI solutions following MLOps best practices.
• Implement model monitoring, performance tracking, data/model drift detection, and automated retraining processes.

Cloud & DevOps

• Design and build cloud-native data and AI solutions on AWS and Azure.
• Develop scalable data platforms using Snowflake, Databricks, and Apache Spark.
• Design and implement data orchestration workflows using Apache Airflow, Azure Data Factory, AWS Glue, or similar orchestration tools.
• Build and maintain automated CI/CD and deployment pipelines using Azure DevOps and GitLab CI/CD.
• Monitor application, data-platform, and pipeline performance and optimize cloud resource utilization, scalability, reliability, and cost.
• Implement cloud security, monitoring, logging, and operational best practices across data and AI platforms.