Data Engineer - AI-Driven Data Pipelines & Lakehouse

Insight Global; Town/Village of Harrison (NY); Vor Ort; USD 120.000 - 160.000; Vollzeit

Zusammenfassung

Insight Global is seeking a hands-on Data Engineer in Harrison, NY to translate evolving R&D requirements into secure, scalable data platforms and pipelines. You will collaborate with AI engineers, data scientists, researchers, software engineers, and product managers to accelerate experimentation and production delivery.

The role involves building scalable batch, streaming, and event‑driven pipelines, using lakehouse and cloud‑native architectures, with emphasis on data quality, security, and

Qualifikationen

• Extensive experience designing and building enterprise‑scale data engineering solutions.
• Strong hands‑on experience with Python, PySpark, SQL, Apache Spark and distributed data processing.
• Experience designing and operating large‑scale batch and real‑time data pipelines and modern lakehouse platforms like databricks.
• Experience building cloud‑native data solutions using AWS, Azure, or comparable enterprise cloud platforms.
• Experience with modern data architecture patterns including data lakes, lakehouses, data products, and event‑driven architectures.

Aufgaben

• Design and build scalable batch, streaming, and event‑driven data pipelines and platforms.
• Develop data ingestion, transformation, enrichment, processing, and consumption capabilities.
• Build reusable data products, APIs, services, and pipelines supporting analytics and AI workloads.
• Implement data governance, quality, lineage, metadata, security, privacy, and access controls.
• Troubleshoot complex data, performance, and production issues.

Kenntnisse

Python

PySpark

SQL

Apache Spark

Data pipelines

Cross-functional collaboration

Tools

Databricks

Airflow

Kafka

MLflow

AWS

Azure

Jobbeschreibung

Insight Global is seeking a hands-on Data Engineer in Harrison, NY to translate evolving R&D requirements into secure, scalable data platforms and pipelines. You will collaborate with AI engineers, data scientists, researchers, software engineers, and product managers to accelerate experimentation and production delivery.

The role involves building scalable batch, streaming, and event‑driven pipelines, using lakehouse and cloud‑native architectures, with emphasis on data quality, security, and