Data Engineer - Microsoft Fabric & Azure Lakehouse

Persistent Systems, LLC; New York (NY); Sur place; USD 133 000 - 166 000; Plein temps

Avantages offerts par ce poste

Medical insurance

Dental insurance

Vision insurance

Life insurance

Disability insurance

Paid time off

401(k) plan with company match

Fitness membership reimbursement

Tuition assistance

Mental health benefits

Pet discounts

Flexible spending accounts

Résumé du poste

Persistent Systems, LLC in New York City is seeking a Data Engineer to design, build, and operate data pipelines, lakehouses, and models for analytics, reporting, and AI workloads with a primary focus on Microsoft Fabric and the Azure data platform.

You will ingest, transform, and publish curated datasets for Power BI, data science, and RAG workloads, ensuring data quality, lineage, and governance while collaborating with analysts, AI engineers, and business stakeholders.

Qualifications

• 3–5 years of professional experience in data engineering or data warehousing.
• Bachelor’s degree in computer science, information systems, engineering, mathematics, or related field.
• Advanced SQL proficiency with joins, CTEs, window functions, MERGE/upsert, and validation.
• Strong Python skills for data engineering, including PySpark or Spark SQL.
• Hands-on experience with Fabric Data Factory or Azure Data Factory and Delta Lake.
• Experience designing lakehouse/data warehouse solutions and medallion architecture.

Responsabilités

• Own data engineering work from discovery through design, build, test, deploy and support.
• Collaborate with analysts, BI developers, AI engineers, and stakeholders to define data requirements and targets.
• Build reusable ingestion patterns, transformation frameworks, and data contracts.
• Evaluate Fabric/Azure tools for scale, security, and total cost of ownership.
• Document data flows, designs, and operating procedures; review peers’ work.
• Design and implement medallion architectures in Fabric Lakehouse/OneLake.
• Develop ingestion with Fabric Data Factory, Azure Data Factory, and Azure Databricks.
• Create transformation logic in PySpark, T-SQL, and KQL; handle quality issues and late data.
• Maintain pipelines with Lakeflow, Unity Catalog, and Git-based CI/CD; monitor performance.
• Ingest streaming data with Fabric Real-Time Intelligence and Azure Event Hubs.
• Model star schemas and collaborate on Power BI semantic models; support Direct Lake.
• Apply data governance, security, and data quality testing; monitor and troubleshoot.
• Prepare datasets for Azure AI Search, Microsoft Foundry, and RAG pipelines.

Connaissances

Advanced SQL

Python for data eng

PySpark / Spark SQL

Data modeling

Delta Lake

Azure Data Factory

Git version control

Azure RBAC / Security

Data quality & lineage

Fabric data tooling

Formation

Bachelor’s degree in CS/IS/Engineering/Math

Outils

Azure Data Factory

Azure Databricks

Unity Catalog

Lakeflow

OneLake

Delta Lake

Description du poste

Persistent Systems, LLC in New York City is seeking a Data Engineer to design, build, and operate data pipelines, lakehouses, and models for analytics, reporting, and AI workloads with a primary focus on Microsoft Fabric and the Azure data platform.

You will ingest, transform, and publish curated datasets for Power BI, data science, and RAG workloads, ensuring data quality, lineage, and governance while collaborating with analysts, AI engineers, and business stakeholders.