Lead Data Engineer

Compunnel, Inc.; Montvale (NJ); On-site; USD 140,000 - 190,000; Full time

Job summary

Compunnel, Inc. is seeking a Lead Data Engineer with 10–12+ years of experience to design, build, and optimize modern enterprise data platforms and AI-enabled data solutions.

This hands-on individual contributor role focuses on scalable data ecosystems, AI-driven data access, and delivering solutions using Microsoft Fabric Data Agent/Fabric IQ or Databricks Genie. The ideal candidate will bring strong expertise in data engineering, cloud platforms, and AI/LLM technologies, collaborating with AI,

Qualifications

• 10–12+ years of professional experience in Data Engineering.
• Strong hands-on expertise with cloud data platforms and enterprise data architectures.
• Experience implementing Microsoft Fabric Data Agent/Fabric IQ or Databricks Genie.
• Knowledge of AI, LLMs, and configuring data agents for AI-powered solutions.
• Experience building data pipelines, ETL/ELT, and data transformation frameworks.
• Experience with data modeling and enterprise analytics platforms.
• Understanding of data governance, security, and performance optimization.
• Ability to drive technical initiatives from design to production.
• Excellent analytical, problem-solving, communication, and collaboration skills.

Responsibilities

• Design, build, and optimize enterprise-scale data platforms and modern data lake architectures.
• Implement, configure, and support Microsoft Fabric Data Agent/Fabric IQ or Databricks Genie solutions.
• Enable AI-driven data access and develop data foundations for Agentic AI use cases.
• Configure and optimize data agents, retrieval systems, and AI-powered data workflows.
• Design, develop, and maintain scalable data pipelines, ingestion frameworks, and ETL/ELT processes.
• Build and optimize data transformation workflows to support analytics and AI initiatives.
• Collaborate with AI, analytics, and business teams to deliver data-driven solutions.
• Ensure data quality, governance, security, scalability, and performance across platforms.
• Evaluate and implement modern data engineering best practices and tools.
• Lead technical solution design, development, testing, and production deployment with minimal supervision.
• Troubleshoot and optimize data platform performance and reliability.

Skills

Analytical skills

Problem-solving

Communication

Collaboration

Tools

Microsoft Fabric Data Agent

Fabric IQ

Databricks Genie

Job description

We are seeking a Lead Data Engineer with 10–12+ years of experience to design, build, and optimize modern enterprise data platforms and AI-enabled data solutions. This hands‑on individual contributor role focuses on developing scalable data ecosystems, enabling AI-driven data access, and delivering intelligent data solutions using Microsoft Fabric Data Agent/Fabric IQ or Databricks Genie. The ideal candidate will possess strong expertise in data engineering, cloud data platforms, and AI/LLM technologies.

Key Responsibilities

• Design, build, and optimize enterprise-scale data platforms and modern data lake architectures.
• Implement, configure, and support Microsoft Fabric Data Agent/Fabric IQ or Databricks Genie solutions.
• Enable AI-driven data access and develop data foundations for Agentic AI use cases.
• Configure and optimize data agents, retrieval systems, and AI-powered data workflows.
• Design, develop, and maintain scalable data pipelines, ingestion frameworks, and ETL/ELT processes.
• Build and optimize data transformation workflows to support enterprise analytics and AI initiatives.
• Collaborate with AI, analytics, and business teams to deliver intelligent, data-driven solutions.
• Ensure data quality, governance, security, scalability, and performance across enterprise data platforms.
• Evaluate and implement modern data engineering best practices, tools, and emerging technologies.
• Lead technical solution design, development, testing, and production deployment with minimal supervision.
• Troubleshoot and optimize data platform performance and reliability.

Required Qualifications

• 10–12+ years of professional experience in Data Engineering.
• Strong hands‑on expertise with modern cloud data platforms and enterprise data architectures.
• Experience implementing Microsoft Fabric Data Agent/Fabric IQ or Databricks Genie.
• Strong knowledge of Artificial Intelligence (AI), Large Language Models (LLMs), and configuring data agents for AI‑powered solutions.
• Experience building or enabling AI agents on enterprise data platforms and data lake environments.
• Strong experience designing and developing data pipelines, ETL/ELT processes, and data transformation frameworks.
• Experience with data modeling and enterprise analytics platforms.
• Strong understanding of enterprise data governance, security, and performance optimization.
• Ability to independently drive technical initiatives from architecture and design through production deployment.
• Excellent analytical, problem‑solving, communication, and collaboration skills.

Preferred Qualifications

• Experience with Microsoft Fabric, Azure Data Services, Databricks, and modern lakehouse architectures.
• Experience with Agentic AI, Retrieval‑Augmented Generation (RAG) architectures, vector databases, or AI‑driven analytics solutions.
• Experience implementing scalable cloud‑native data engineering solutions and enterprise AI platforms.