Data Engineer: Real-Time IoT & Lakehouse Pipelines

Gridware; San Francisco (CA); On-site; USD 120,000 - 190,000; Full time

Benefits offered by this job

Health, Dental & Vision plans

Paid parental leave

Alternating day off

Off the Grid – two weeks paid break

Commuter allowance

Company‑paid training

Job summary

Gridware is seeking a Data Engineer to build and maintain pipelines powering our Active Grid Response platform. You’ll work with cross‑functional teams to ensure telemetry and sensor data flow reliably from ingestion to analytics and monitoring tools.

In this hands‑on role, you’ll deepen expertise in distributed data systems, focusing on data quality, governance, and scalable architectures in a fast‑growth environment.

Qualifications

• 2–4 years of experience as a Data Engineer or Backend Engineer with heavy data exposure.
• Proficiency in Python and SQL.
• Familiarity with data warehouses, Lakehouse platforms, or big data tools (Databricks, Spark).
• Experience with pipeline orchestration tools (Airflow, Dagster, Prefect, etc.).
• Understanding of event‑driven systems or streaming platforms (Kafka, Kinesis, Pub/Sub).
• Solid foundation in data modeling, testing, and version control.
• Ability to work collaboratively in a high‑autonomy, fast‑paced environment.
• Experience with IoT, telemetry ingestion, or time‑series data.
• Exposure to Unity Catalog, governance, or schema enforcement.
• Understanding of Protobuf, Avro, Parquet, or serialization formats.
• Hands‑on experience with observability tools (Grafana, OpenTelemetry).

Responsibilities

• Building ETL/ELT pipelines that ingest transformer, pole, and sensor telemetry into Gridware’s Data Lake and Lakehouse.
• Developing and maintaining real‑time and batch ingestion processes using Python, SQL, Databricks, and Spark.
• Implementing data quality checks, validation rules, and automated testing for stable operations.
• Collaborating with Software, Firmware, and Data Science teams to define ingestion schemas and transformations.
• Working with cloud‑native tools to optimize pipeline throughput and cost efficiency.
• Monitoring pipelines for reliability, troubleshooting issues, and contributing to on‑call rotations.
• Writing documentation for data processes, models, and metadata.

Skills

Python

SQL

Data modeling

Data quality

Version control

Automation

Communication

IoT data

Tools

Databricks

Spark

Airflow

Dagster

Prefect

Kafka

Kinesis

Unity Catalog

Job description

Gridware is seeking a Data Engineer to build and maintain pipelines powering our Active Grid Response platform. You’ll work with cross‑functional teams to ensure telemetry and sensor data flow reliably from ingestion to analytics and monitoring tools.

In this hands‑on role, you’ll deepen expertise in distributed data systems, focusing on data quality, governance, and scalable architectures in a fast‑growth environment.