Senior Platform Data Engineer — Real-Time AI/IoT Pipelines

BrightAI; Palo Alto (CA); On-site; USD 150,000 - 210,000; Full time

Job summary

BrightAI is hiring a Platform Data Engineer in Palo Alto to design and maintain scalable data ingestion pipelines for massive sensor and operational data streams across drones, robotics, industrial systems and smart environments.

You will build secure data lake and Delta Lake-based storage, implement end-to-end data workflows in AWS, and collaborate with Cloud, Edge, Product, and AI teams to support ML development and model serving with near real-time predictions.

Qualifications

• 5+ years of professional experience in software or data engineering.
• Strong programming skills in Python and Golang.
• Deep understanding of SQL and modern data lake architectures (Parquet, Iceberg, Delta Lake).
• Hands-on experience with AWS services: S3, Lambda, Glue, Kinesis, Athena, RDS.
• Proficiency with Terraform for automating infrastructure deployment.
• Hands-on experience with Databricks for ML workflows and Delta Lake.
• Experience with real-time or batch data ingestion at scale and fault-tolerant ETL/ELT design.
• Familiarity with Kafka or Kinesis for event-driven architectures.

Responsibilities

• Design, build, and maintain scalable data ingestion pipelines for structured and semi-structured data.
• Implement secure data lake and SQL-based storage architectures optimized for performance and cost.
• Develop and maintain internal data ingestion tools using Python, Golang, and SQL.
• Build and maintain inference pipelines to support model serving and near real-time predictions.
• Support ML development workflow, including experimentation, testing, deployment and monitoring.
• Collaborate with Cloud, Edge, Product, and AI teams to define data contracts, schemas, and retention policies.
• Utilize AWS infrastructure (S3, Lambda, Glue, Kinesis, Athena, RDS) to support end-to-end data workflows.
• Use Terraform to manage data platform infrastructure.
• Monitor data pipelines for quality, latency, and failures using CloudWatch, DataDog, or similar tools.
• Land curated sensor/operational data into Delta tables on Databricks to support ML experiments.
• Optimize storage, partitioning, and query performance across large datasets.
• Participate in architecture reviews ensuring security and compliance.

Skills

Python

Golang

SQL

AWS

Terraform

Databricks

ML pipelines

Data ingestion

Kafka/Kinesis

Debugging

Tools

Parquet/Delta Lake

Kinesis

Spark SQL

Iceberg

Job description

BrightAI is hiring a Platform Data Engineer in Palo Alto to design and maintain scalable data ingestion pipelines for massive sensor and operational data streams across drones, robotics, industrial systems and smart environments.

You will build secure data lake and Delta Lake-based storage, implement end-to-end data workflows in AWS, and collaborate with Cloud, Edge, Product, and AI teams to support ML development and model serving with near real-time predictions.