Data Engineer

Insight Global; Town/Village of Harrison (NY); On-site; USD 120,000 - 160,000; Full time

Job summary

Insight Global is seeking a hands-on Data Engineer in Harrison, NY to translate evolving R&D requirements into secure, scalable data platforms and pipelines. You will collaborate with AI engineers, data scientists, researchers, software engineers, and product managers to accelerate experimentation and production delivery.

The role involves building scalable batch, streaming, and event‑driven pipelines, using lakehouse and cloud‑native architectures, with emphasis on data quality, security, and

Qualifications

• Extensive experience designing and building enterprise‑scale data engineering solutions.
• Strong hands‑on experience with Python, PySpark, SQL, Apache Spark and distributed data processing.
• Experience designing and operating large‑scale batch and real‑time data pipelines and modern lakehouse platforms like databricks.
• Experience building cloud‑native data solutions using AWS, Azure, or comparable enterprise cloud platforms.
• Experience with modern data architecture patterns including data lakes, lakehouses, data products, and event‑driven architectures.

Responsibilities

• Design and build scalable batch, streaming, and event‑driven data pipelines and platforms.
• Develop data ingestion, transformation, enrichment, processing, and consumption capabilities.
• Build reusable data products, APIs, services, and pipelines supporting analytics and AI workloads.
• Implement data governance, quality, lineage, metadata, security, privacy, and access controls.
• Troubleshoot complex data, performance, and production issues.

Skills

Python

PySpark

SQL

Apache Spark

Data pipelines

Cross-functional collaboration

Tools

Databricks

Airflow

Kafka

MLflow

AWS

Azure

Job description

Job Description

This is a highly hands‑on engineering role focused on translating evolving R&D requirements into secure, reliable, scalable, and production‑ready data platforms and pipelines. You will collaborate with AI Engineers, Data Scientists, Researchers, Software Engineers, and Product Managers to accelerate experimentation and production delivery.

• Design and build scalable batch, streaming, and event‑driven data pipelines and platforms.
• Develop data ingestion, transformation, enrichment, processing, and consumption capabilities using modern lakehouse and cloud‑native architectures.
• Build reusable data products, APIs, services, and pipelines supporting analytics, machine learning, generative AI, and agentic AI workloads.
• Optimize solutions for performance, scalability, reliability, resiliency, observability, and cost efficiency.
• Implement engineering practices including data modeling, automated testing, CI/CD, Infrastructure as Code, and deployment automation.
• Implement data quality, lineage, metadata, cataloging, security, privacy, governance, and access controls.
• Evaluate emerging data and AI technologies through prototypes and proof‑of‑concepts and transition successful solutions toward production.
• Troubleshoot complex data, performance, scalability, and production issues.
• Translate evolving product and R&D requirements into practical technical designs and incremental delivery plans.
• Collaborate with cross‑functional teams and communicate technical decisions, dependencies, risks, and trade‑offs.

Skills and Requirements

• Extensive experience designing and building enterprise‑scale data engineering solutions.
• Strong hands‑on experience with Python, PySpark, SQL, Apache Spark and distributed data processing.
• Experience designing and operating large‑scale batch and real‑time data pipelines and modern lakehouse platforms like databricks.
• Experience building cloud‑native data solutions using AWS, Azure, or comparable enterprise cloud platforms.
• Experience with modern data architecture patterns including data lakes, lakehouses, data products, and event‑driven architectures.
• Experience with orchestration technologies such as Apache Airflow, Databricks Workflows equivalent.
• Strong understanding of data governance, data quality, lineage, metadata management, security, privacy, and access controls.
• Ability to diagnose and resolve complex performance, scalability, reliability, and operational issues.
• Strong problem‑solving, communication, and cross‑functional collaboration skills.
• Experience integrating structured, semi‑structured, unstructured, and streaming data.
• Ability to work effectively in an R&D environment with evolving requirements and rapid experimentation.
• Experience supporting machine learning, generative AI, or agentic AI applications.
• Experience with CI/CD, automated testing, version control, Infrastructure as Code, observability, and platform automation.
• Experience with MLflow, Azure Machine Learning, Azure AI Foundry, or Microsoft Fabric.
• Experience with Kafka or other event‑streaming technologies.
• Experience in payments, financial services, or another regulated industry.

We are a company committed to creating diverse and inclusive environments where people can bring their full, authentic selves to work every day. We are an equal employment opportunity/affirmative action employer that believes everyone matters. Qualified candidates will receive consideration for employment without regard to race, color, ethnicity, religion, sex (including pregnancy), sexual orientation, gender identity and expression, marital status, national origin, ancestry, genetic factors, age, disability, protected veteran status, military or uniformed service member status, or any other status or characteristic protected by applicable laws, regulations, and ordinances. If you need assistance and/or a reasonable accommodation due to a disability during the application or the recruiting process, please send a request to [e-mail].