Senior Spark & Data Platform Engineer (Remote)

Cephas Consultancy Services Private Limited; Georgia; Híbrido; USD 120.000 - 190.000; A tiempo parcial

Descripción de la vacante

Cephas Consultancy Services Private Limited is seeking a Sr. Software Development Engineer to join its remote-ready team serving Georgia, United States.

You will build and operate production Spark pipelines, design data processing models, and run REST APIs on AWS with a focus on reliability and security. The role demands deep JVM/Python experience, SQL, and familiarity with Snowflake/ Iceberg, Airflow, and AI-assisted development tools; candidate should thrive in a fast-moving, AI-forward

Formación

• 7+ years of professional software or data engineering experience.
• 4+ years building, optimizing, and maintaining production Apache Spark pipelines, with strong Java and Python skills across JVM Spark and PySpark.
• Advanced SQL, relational data modeling, and large-scale data processing experience, including production work with Snowflake, MySQL, and Iceberg.
• 3+ years operating AWS data workloads using S3 and Airflow (or equivalent production workflow orchestration); container orchestration and data-catalog experience a plus.
• 3+ years building, changing, and operating production REST APIs, including PHP APIs built with Symfony or a closely equivalent PHP MVC framework.
• Demonstrated proficiency using AI development tools to deliver production-quality work efficiently — including AI-assisted coding, code review, documentation, and investigation, working with agentic code harnesses, and spec-based (spec-driven) AI development — with the judgment to know when to rely on AI output and when not to.
• Ability to independently deliver secure, backward-compatible API changes, including implementation, validation, authorization, data access, documentation, automated testing, and production diagnostics.
• Experience implementing external provider or client integrations involving schemas, APIs, file delivery, identity fields, reconciliation, privacy-sensitive data, and SLAs.
• Experience writing unit and integration tests for data pipelines, APIs, and web applications.
• Ability to diagnose production failures, reconcile data, execute backfills, manage staged releases and rollbacks, and support delivery commitments.
• Hands-on experience building and running big data systems, with a focus on performance, reliability, and data quality.
• Experience in ad tech or advertising measurement, such as attribution, conversions, audience and impression data, identity matching, or partner optimization feeds.
• Ability to ramp quickly and deliver with minimal onboarding in an existing, complex codebase.

Responsabilidades

• Build, optimize, and operate production Apache Spark pipelines that generate attributed conversion and optimization feeds from large-scale impression and conversion data.
• Design and evolve data processing and models across modern data lake and warehouse technologies, orchestrated with a production workflow scheduler on AWS.
• Build and operate partner optimization feeds and integrations with external partners, including schemas, identity fields and unique IDs, file delivery, reconciliation, and SLAs.
• Build, change, and operate production REST APIs, including PHP APIs, delivering secure, backward-compatible changes end to end (implementation, validation, authorization, data access, documentation, automated testing, and production diagnostics).
• Diagnose production failures, reconcile partner-facing data, execute backfills, and manage staged releases and rollbacks.
• Write and maintain unit and integration tests across pipelines, APIs, and feeds.

Descripción del empleo

Cephas Consultancy Services Private Limited is seeking a Sr. Software Development Engineer to join its remote-ready team serving Georgia, United States.

You will build and operate production Spark pipelines, design data processing models, and run REST APIs on AWS with a focus on reliability and security. The role demands deep JVM/Python experience, SQL, and familiarity with Snowflake/ Iceberg, Airflow, and AI-assisted development tools; candidate should thrive in a fast-moving, AI-forward