Senior Data Engineer — Python, Spark & Databricks (Remote)

Principal33; United States; Remote; USD 120,000 - 180,000; Full time

Benefits offered by this job

Medical insurance

Holiday flat in Valencia

Gifts for special occasions

Anniversary gifts

Team-building events

End of year celebrations

Day off on your birthday

Access to Udemy

German language courses

Personal and professional growth

Job summary

principal33 is seeking an experienced Data Engineer to design, build and operate scalable batch, micro-batch and streaming pipelines using Python, PySpark and Azure Databricks.

You will integrate diverse data sources including REST APIs, GraphQL, WebSocket and gRPC, develop robust ingestion for JSON, Parquet and Delta, and implement reliable data quality and governance within a remote-friendly environment.

Qualifications

• 5+ years of data engineering experience.
• Strong Python and PySpark skills with production pipelines.
• Experience with Azure Databricks and cloud data platforms.
• Design idempotent, back-fillable data processes and pipelines.
• Knowledge of data modeling, Lakehouse/Medallion architectures.
• Familiarity with data quality, lineage and governance concepts.
• Experience with CI/CD and code reviews.

Responsibilities

• Design and operate scalable batch, micro-batch and streaming pipelines.
• Integrate internal and external data sources via APIs, GraphQL, WebSocket and gRPC.
• Develop robust ingestion for JSON, CSV, Parquet, Delta and APIs.
• Implement pagination, retries, backoff and checkpointing.
• Ensure idempotent processing and graceful backfills.
• Collaborate with traders, analysts, and platform teams.

Skills

Python

PySpark

Azure Databricks

Tools

REST APIs

GraphQL

WebSocket

gRPC

Job description

principal33 is seeking an experienced Data Engineer to design, build and operate scalable batch, micro-batch and streaming pipelines using Python, PySpark and Azure Databricks.

You will integrate diverse data sources including REST APIs, GraphQL, WebSocket and gRPC, develop robust ingestion for JSON, Parquet and Delta, and implement reliable data quality and governance within a remote-friendly environment.