Remote Data Platform Engineer - Spark, Databricks

CyberSearch; New York (NY); Hybrid; USD 257,611,000 - 371,952,000; Full time

Job summary

CyberSearch is seeking a Data Platform Engineer to build data infrastructure for AI/ML and product teams. The role emphasizes Apache Spark, Databricks, Python and/or Scala, and distributed data processing.

You will develop, operate, and scale pipelines across cloud environments while ensuring data quality and reliability. The ideal candidate designs end-to-end data platforms, contributes across the data stack, and collaborates with teams to translate requirements into production-ready data

Qualifications

• Hands-on experience building production data pipelines with Spark and Databricks.
• Strong Python and/or Scala development for data engineering and distributed processing.
• Design, build, deploy, and maintain large-scale data pipelines.
• Experience building CI/CD pipelines and automated workflows (build/test/deploy).
• Experience with AWS, Azure, or GCP in production environments.
• Implement data transformation, validation, monitoring, and data quality controls for pipelines.
• Breadth of contribution across the data platform, not narrow specialization.
• Troubleshooting distributed data workloads for performance and scalability.
• Healthcare data experience (RCM, claims, encounters) or regulated data environments.
• Proactive, fast-moving with production-ready pipeline solutions.

Skills

Apache Spark

Databricks

Python

Scala

CI/CD

AWS

Azure

GCP

Data pipelines

Data quality

Distributed processing

Data validation

Monitoring

Healthcare data

Job description

CyberSearch is seeking a Data Platform Engineer to build data infrastructure for AI/ML and product teams. The role emphasizes Apache Spark, Databricks, Python and/or Scala, and distributed data processing.

You will develop, operate, and scale pipelines across cloud environments while ensuring data quality and reliability. The ideal candidate designs end-to-end data platforms, contributes across the data stack, and collaborates with teams to translate requirements into production-ready data