Data Engineer | C2H | Face To Face Interview

Virtues R & D; United States; A distancia; USD 120.000 - 190.000; Jornada completa

Descripción de la vacante

Virtues R & D is seeking a Data Engineer specialized in Microsoft Fabric, batch and streaming pipelines. The role drives data pipelines from design through production, with emphasis on scalable batch and streaming workflows and cost-conscious optimization.

You will own data ingestion, transformation, and modeling across diverse sources, mentoring junior engineers and collaborating with product teams in an agile environment.

Formación

• Proficiency in Python, Java, or Scala for building data pipelines.
• Hands-on with Spark, Kafka, Flink, or dbt; at least two in use.
• Strong SQL and NoSQL skills; solid data modeling fundamentals.
• Experience with Microsoft Fabric Notebooks in real projects.
• Docker and Kubernetes proficiency for containerized pipelines.
• Observability tooling experience; Datadog preferred.
• SQL and pipeline optimization for speed and cost.

Responsabilidades

• Develop robust batch and streaming pipelines from API, DB, event streams, and vendor systems.
• Create reusable ingestion, transformation, and storage patterns; model data into trusted datasets.
• Build and troubleshoot workflows in Microsoft Fabric Notebooks and move to production.
• Implement automated tests, version control, and CI/CD for pipelines.
• Set up data quality checks, alerting, and monitoring to surface issues early.
• Tune queries, pipelines, and storage to manage runtime and cloud spend.
• Maintain documentation of data flows, lineage, and metadata.
• Collaborate in agile ceremonies with product managers, developers, and analysts.
• Mentor junior engineers and ensure data privacy and security standards.

Conocimientos

Python

Java

Scala

Spark

Kafka

SQL

NoSQL

Microsoft Fabric Notebooks

Docker

Kubernetes

Datadog

SQL optimization

Fabric Lakehouse

Warehouse

Data Pipelines

Dataflows Gen2

Semantic Models

Azure/AWS data services

CI/CD (GitHub Actions)

AI tools for speed

Herramientas

Datadog

Docker

Kubernetes

Descripción del empleo

Data Engineer: Microsoft Fabric, Batch and Streaming Pipelines

Experience: 5+ years (adjust to the client's level) | Hours: Full-time, 40 hrs/week, aligned to client's team time zone

Role Overview
Join a product-focused team to build and run the data pipelines behind customer-facing applications and analytics. You'll own your pipelines from first design through production support, with a focus on Microsoft Fabric.

Key Responsibilities

• Develop robust batch and streaming pipelines that pull data from APIs, databases, event streams, and external vendor systems.
• Create reusable patterns for ingesting, transforming, and storing data, and model it into consistent, trusted datasets.
• Build and troubleshoot workflows in Microsoft Fabric Notebooks and move them into production.
• Put automated testing, version control, and CI/CD around every pipeline you ship.
• Set up data quality checks, alerting, and monitoring so issues surface before users notice them.
• Tune queries, pipelines, and storage to keep runtime and cloud spend under control.
• Maintain clear documentation of data flows, lineage, and metadata.
• Work within agile ceremonies (planning, standups, retros) alongside product managers, developers, and analysts.
• Follow data privacy and security standards, and help mentor junior engineers.

Mandatory Skills

• Programming: Python, Java, or Scala for pipeline development
• Data Frameworks: Hands-on with at least two of Spark, Kafka, Flink, dbt
• Databases: Strong SQL plus working knowledge of NoSQL; solid data modeling fundamentals
• Platform: Microsoft Fabric Notebooks, used in real project work
• Containers & Orchestration: Docker and Kubernetes
• Monitoring: Experience with observability tooling; Datadog strongly preferred
• Performance: Proven SQL and pipeline optimization for speed and cost
• Good to have: Fabric Lakehouse, Warehouse, Data Pipelines, Dataflows Gen2, Semantic Models; Azure or AWS data services; CI/CD tools (GitHub Actions, Jenkins, Azure DevOps); use of generative or agentic AI tools to speed up engineering work