Data Engineer: Batch & Streaming Pipelines on Fabric

Virtues R & D; United States; Remote; USD 120,000 - 190,000; Full time

Job summary

Virtues R & D is seeking a Data Engineer specialized in Microsoft Fabric, batch and streaming pipelines. The role drives data pipelines from design through production, with emphasis on scalable batch and streaming workflows and cost-conscious optimization.

You will own data ingestion, transformation, and modeling across diverse sources, mentoring junior engineers and collaborating with product teams in an agile environment.

Qualifications

• Proficiency in Python, Java, or Scala for building data pipelines.
• Hands-on with Spark, Kafka, Flink, or dbt; at least two in use.
• Strong SQL and NoSQL skills; solid data modeling fundamentals.
• Experience with Microsoft Fabric Notebooks in real projects.
• Docker and Kubernetes proficiency for containerized pipelines.
• Observability tooling experience; Datadog preferred.
• SQL and pipeline optimization for speed and cost.

Responsibilities

• Develop robust batch and streaming pipelines from API, DB, event streams, and vendor systems.
• Create reusable ingestion, transformation, and storage patterns; model data into trusted datasets.
• Build and troubleshoot workflows in Microsoft Fabric Notebooks and move to production.
• Implement automated tests, version control, and CI/CD for pipelines.
• Set up data quality checks, alerting, and monitoring to surface issues early.
• Tune queries, pipelines, and storage to manage runtime and cloud spend.
• Maintain documentation of data flows, lineage, and metadata.
• Collaborate in agile ceremonies with product managers, developers, and analysts.
• Mentor junior engineers and ensure data privacy and security standards.

Skills

Python

Java

Scala

Spark

Kafka

SQL

NoSQL

Microsoft Fabric Notebooks

Docker

Kubernetes

Datadog

SQL optimization

Fabric Lakehouse

Warehouse

Data Pipelines

Dataflows Gen2

Semantic Models

Azure/AWS data services

CI/CD (GitHub Actions)

AI tools for speed

Tools

Datadog

Docker

Kubernetes

Job description

Virtues R & D is seeking a Data Engineer specialized in Microsoft Fabric, batch and streaming pipelines. The role drives data pipelines from design through production, with emphasis on scalable batch and streaming workflows and cost-conscious optimization.

You will own data ingestion, transformation, and modeling across diverse sources, mentoring junior engineers and collaborating with product teams in an agile environment.