Data Engineer

CyberSearch; New York (NY); Hybrid; USD 257,611,000 - 371,952,000; Full time

Job summary

CyberSearch is seeking a Data Platform Engineer to build data infrastructure for AI/ML and product teams. The role emphasizes Apache Spark, Databricks, Python and/or Scala, and distributed data processing.

You will develop, operate, and scale pipelines across cloud environments while ensuring data quality and reliability. The ideal candidate designs end-to-end data platforms, contributes across the data stack, and collaborates with teams to translate requirements into production-ready data

Qualifications

• Hands-on experience building production data pipelines with Spark and Databricks.
• Strong Python and/or Scala development for data engineering and distributed processing.
• Design, build, deploy, and maintain large-scale data pipelines.
• Experience building CI/CD pipelines and automated workflows (build/test/deploy).
• Experience with AWS, Azure, or GCP in production environments.
• Implement data transformation, validation, monitoring, and data quality controls for pipelines.
• Breadth of contribution across the data platform, not narrow specialization.
• Troubleshooting distributed data workloads for performance and scalability.
• Healthcare data experience (RCM, claims, encounters) or regulated data environments.
• Proactive, fast-moving with production-ready pipeline solutions.

Skills

Apache Spark

Databricks

Python

Scala

CI/CD

AWS

Azure

GCP

Data pipelines

Data quality

Distributed processing

Data validation

Monitoring

Healthcare data

Job description

{REMOTE-Hybrid} primary with commutable distance to a hub

RATE $90 to 130.00hr

*** must be a US citizen or green card ***

Overview:

We are seeking a Data Platform Engineer who can build the data infrastructure that transforms large-scale healthcare data into reliable, production-ready datasets for AI/ML and product teams. This is a hands-on data engineering role with a strong emphasis on Apache Spark, Databricks, and distributed data processing. You will work across Spark, Databricks, Python and/or Scala, CI/CD, cloud infrastructure, and data quality to build and operate pipelines at scale. The ideal candidate is not simply focused on individual data transformations—they understand how data moves through a platform, how pipelines perform at scale, and how engineering decisions translate into reliable, high-quality data products.

Requirements:

• Must have demonstrated hands-on experience building production data pipelines using Apache Spark and Databricks.
• Must have strong Python and/or Scala development experience for data engineering and distributed processing.
• Proven ability to design, build, deploy, and maintain large-scale data pipelines processing complex datasets.
• Demonstrated experience building CI/CD pipelines and automated engineering workflows, including build, testing, deployment, and staging processes.
• Demonstrated experience working with AWS, Azure, or GCP in production data engineering environments.
• Proven ability to implement data transformation, validation, monitoring, and data quality controls for production pipelines.
• Demonstrated breadth of contribution across the data platform rather than deep specialization in only one narrow technical area.
• Experience troubleshooting and optimizing distributed data workloads for performance, reliability, scalability, and throughput.
• Experience in healthcare data, revenue cycle management (RCM), claims, patient encounters, clinical data, or other regulated/high-accuracy data environments.
• Strong builder and problem-solving mindset; comfortable moving quickly from requirements or data problems to production-ready pipeline solutions.