Data Engineer

Virtues R & D; United States; Remote; USD 140,000 - 190,000; Full time

Job summary

Virtues R & D is seeking a Data Engineer to own batch and streaming pipelines, with a focus on Microsoft Fabric Notebooks. You will design, build, and operate data pipelines behind analytics products, collaborating with product managers and analysts.

You will implement robust ingestion patterns, optimize performance and cost, apply testing and CI/CD, and mentor junior engineers while upholding data privacy and security standards in a fast-paced, product-driven environment.

Qualifications

• Proficient in Python, Java, or Scala for pipeline development.
• Experience with two or more data frameworks (Spark, Kafka, Flink, dbt).
• Strong SQL with NoSQL knowledge and data modeling fundamentals.
• Experience with Microsoft Fabric Notebooks in production.
• Familiar with Docker and Kubernetes for containerized workloads.
• Experience with observability tooling (Datadog preferred).
• SQL optimization for performance and cost; data pipeline tuning.
• Knowledge of data privacy and security standards; mentor junior engineers (nice to have).

Responsibilities

• Develop robust batch and streaming pipelines from APIs, databases, event streams, and external vendor systems.
• Create reusable ingestion, transformation, and storage patterns and model data into trusted datasets.
• Build and productionize workflows in Microsoft Fabric Notebooks.
• Apply automated testing, version control, and CI/CD around pipelines.
• Set up data quality checks, alerting, and monitoring to surface issues.
• Tune queries, pipelines, and storage to control runtime and cloud spend.
• Maintain documentation of data flows, lineage, and metadata.
• Collaborate in Agile ceremonies with product managers, developers, and analysts.
• Mentor junior engineers and uphold data privacy standards.

Skills

Python

Java

Scala

Tools

Spark

Kafka

Flink

dbt

Job description

Data Engineer: Microsoft Fabric, Batch and Streaming Pipelines

Experience: 5+ years (adjust to the client's level) | Hours: Full-time, 40 hrs/week, aligned to client's team time zone

Role Overview
Join a product-focused team to build and run the data pipelines behind customer-facing applications and analytics. You'll own your pipelines from first design through production support, with a focus on Microsoft Fabric.

Key Responsibilities

• Develop robust batch and streaming pipelines that pull data from APIs, databases, event streams, and external vendor systems.
• Create reusable patterns for ingesting, transforming, and storing data, and model it into consistent, trusted datasets.
• Build and troubleshoot workflows in Microsoft Fabric Notebooks and move them into production.
• Put automated testing, version control, and CI/CD around every pipeline you ship.
• Set up data quality checks, alerting, and monitoring so issues surface before users notice them.
• Tune queries, pipelines, and storage to keep runtime and cloud spend under control.
• Maintain clear documentation of data flows, lineage, and metadata.
• Work within agile ceremonies (planning, standups, retros) alongside product managers, developers, and analysts.
• Follow data privacy and security standards, and help mentor junior engineers.

Mandatory Skills

• Programming: Python, Java, or Scala for pipeline development
• Data Frameworks: Hands-on with at least two of Spark, Kafka, Flink, dbt
• Databases: Strong SQL plus working knowledge of NoSQL; solid data modeling fundamentals
• Platform: Microsoft Fabric Notebooks, used in real project work
• Containers & Orchestration: Docker and Kubernetes
• Monitoring: Experience with observability tooling; Datadog strongly preferred
• Performance: Proven SQL and pipeline optimization for speed and cost
• Good to have: Fabric Lakehouse, Warehouse, Data Pipelines, Dataflows Gen2, Semantic Models; Azure or AWS data services; CI/CD tools (GitHub Actions, Jenkins, Azure DevOps); use of generative or agentic AI tools to speed up engineering work