Lead Data Engineer

Straive; Newark (NJ); On-site; USD 140,000 - 200,000; Full time

Job summary

Straive is seeking a Lead Data Engineer to architect and implement modern data solutions within the Microsoft Fabric ecosystem in Newark, NJ. You will streamline data workflows, enable enterprise data integration, and drive analytics initiatives across the organization.

Responsibilities cover OneLake storage, unified data pipelines, Bronze/Silver/Gold lakehouse architectures, and advanced optimizations in PySpark, SQL, and Delta Lake.

Qualifications

• 10-15 years total experience in data engineering or similar roles.
• 3-5 years Azure Data Engineering focusing on Microsoft Fabric, Synapse Analytics, and OneLake.
• Advanced Python (PySpark) and SQL proficiency.
• Hands-on with Delta Lake formats, Medallion Architecture, and Spark-based processing.
• Experience with Power BI Direct Lake connectivity, Git-based CI/CD, and Fabric Capacity Management.
• Proficient in building and monitoring pipelines in Data Factory and managing CI/CD via Azure DevOps.
• Understanding of Star/Snowflake data modeling and semantic modeling.
• Preferred certifications: DP-600, DP-203, DP-700.

Responsibilities

• Unified Storage: implement OneLake structures as single source of truth for varied data assets.
• Data Engineering: develop scalable ETL/ELT via Synapse Data Engineering and Data Factory.
• Performance Tuning: optimize Fabric capacity and compute for cost and speed.
• Pipelines & Ingestion: build metadata-driven pipelines using Fabric Pipelines; handle JSON/ZIP/XLSX/CSV; implement incremental refreshes and SCD logic.
• Architecture & Performance: design Bronze/Silver/Gold lakehouses and optimize with PySpark, SQL, and Delta Lake features.
• Automation & Scripting: write Python and use Fabric REST APIs to automate notebook workflows and workspace management.
• Governance & Data Quality: automate validation, lineage, RLS/CLS controls, and CICD-driven environments.
• DevOps, Capacity & Analytics: manage DevOps/ALM, monitor compute, and support Direct Lake BI and real‑time ingestion.
• Collaboration: work with Data Scientists and BI teams for GenAI-ready datasets.

Skills

Python (PySpark)

SQL

Delta Lake

Azure Data Engineering

Microsoft Fabric

Synapse Analytics

OneLake

Power BI Direct Lake

Git CI/CD

Fabric Capacity Management

Education

DP-600 Fabric Analytics Engineer

DP-203 Azure Data Engineer

DP-700 Fabric Data Engineer Associate

Tools

Data Factory

Azure DevOps

Fabric REST APIs

Fabric Utils

Job description

We are seeking a Lead Data Engineer to help us architect and implement modern data solutions using the Microsoft Fabric ecosystem. You will be responsible for streamlining data workflows, implementing OneLake architectures, and enabling seamless data integration and analytics across the enterprise.

Key Responsibilities:

• Unified Storage: Implement and optimize OneLake structures to create a seamless, single source of truth for diverse data assets.
• Data Engineering: Develop scalable ETL/ELT workflows using Synapse Data Engineering and Data Factory.
• Performance Tuning: Optimize Fabric capacity and compute usage to ensure cost-efficient, high-speed data processing.
• Pipelines & Ingestion: Build metadata-driven pipelines using Fabric Pipelines to ingest and process JSON, ZIP, XLSX, and CSV files effectively. Implement pattern-based incremental refreshes and SCD Type 1 & Type 2 logic.
• Architecture & Performance: Design scalable Medallion Architectures (Bronze/Silver/Gold) across Fabric Lakehouses and Data Warehouses. Apply PySpark optimization (broadcast joins, partition strategy, data skew, memory tuning),SQL query tuning, and Delta Lake features (V-Order, Liquid Clustering, Time Travel).
• Automation & Scripting: Write advanced Python and leverage Fabric Utils (notebookutils/mssparkutils) alongside Fabric REST APIs to automate inter-notebook workflows, dynamic parameterization, and workspace management.
• Governance & Data Quality: Build automated Data Quality practices (schema enforcement, dynamic validation, reconciliation logging). Implement Fabric Governance frameworks, Microsoft Purview integration, data lineage, and security controls(RLS/CLS). Managing staged environments Dev. UAT and PROD with CICD Principals.
• DevOps, Capacity & Analytics: Manage Fabric DevOps/ALM (Git integration and deployment pipelines) and monitor compute using the Fabric Capacity Metrics App to optimize CUs. Support basic Power BI semantic modeling (Direct Lake mode) and real-time ingestion via Eventhouse/KQL.
• Collaboration: Partner with Data Scientists and BI teams to deliver structured, high-fidelity datasets for GenAI and advanced analytics.

Key Qualifications & Skills

• Total Experience of 10-15 years
• Ecosystem Mastery: 3-5 years inAzure Data Engineering with focused experience in Microsoft Fabric, Synapse Analytics, and OneLake.
• Languages: Advanced proficiency in Python (PySpark) and Complex SQL.
• Frameworks: Hands-on experience with Delta Lake formats, Medallion Architecture, and Spark-based processing.
• Integrations: Working knowledge of Power BI (Direct Lake connectivity), Git-based CI/CD, and Fabric Capacity Management.
• Orchestration: Proven ability to build and monitor pipelines in Data Factory and manage CI/CD via Azure DevOps.
• Data Modeling: Good understanding of Star/Snowflake schemas and semantic modeling.
• Certifications (Preferred): DP-600 (Fabric Analytics Engineer) or DP-203 (Azure Data Engineer), DP-700 (Fabric Data Engineer Associate).

WHY STRAIVE:

Straive is a global leader in enterprise-grade data analytics and AI solutions, committed to empowering businesses across various industries with cutting-edge technology and expert insights. Backed by EQT, a top private equity firm, we are uniquely positioned to drive innovation through significant investments and an entrepreneurial spirit.

Our core focus is on delivering advanced Data Analytics & AI Solutions. By combining sophisticated technology with subject matter expertise, we deliver material impact on our clients' topline and streamline their operations.

With a client base spanning 30 countries, Straive's strategically located teams operate from eight countries and is headquartered in Singapore. This global presence enables us to offer localized expertise with a worldwide perspective.

Join Straive to be part of a dynamic team at the forefront of data analytics and AI innovation. Here, you'll have the opportunity to contribute to transformative projects, supported by significant investments and an entrepreneurial drive fueled by our partnership with EQT.

Recognition & Achievements

We have been recognized as a Star Performer in Data & AI Services Specialists - Everest’s

• North America PEAK Matrix 2025, and as a Leader in AIM’s Pema Quadrant of Agentic AI Service Providers - 2025.
• In Nov 2023, Straive acquired Gramener, an award-winning, design-led data science company, enhancing our data, analytics, and AI capabilities.
• In June 2025, we acquired SG Analytics, a leading provider of AI-powered insights and contextual analytics services.

For more information, please visit our Website https://www.straive.com/

“Straive is an equal opportunity employer. We celebrate diversity and are committed to creating an inclusive environment for all employees. All employment decisions are based on business needs, job requirements, and individual qualifications, without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, veteran, or disability status”