Senior Big Data Engineer: Scalable Pipelines & Low Latency

Remote; San Francisco (CA); Presencial; USD 150.000 - 210.000; Jornada completa

Descripción de la vacante

BrightEdge, a leading SEO and content performance platform, is seeking a Big Data Engineer to design and maintain high-performance data pipelines.

You will optimize batch processing, tackle latency bottlenecks, build robust ETL workflows, and collaborate with data scientists to scale our data infrastructure.

The role requires 4+ years in data engineering, Python expertise, and hands-on Spark/Hadoop, Airflow, and SQL/NoSQL experience.

Formación

• Bachelor's degree in Computer Science, Engineering, or related field.
• 4+ years of experience in data engineering roles.
• Strong Python programming skills for data processing.
• Experience with Spark, Hadoop, and big data technologies.
• Proven experience optimizing database performance (SQL/NoSQL).
• Knowledge of data pipeline orchestration tools (Airflow, Luigi).
• Understanding of performance optimization techniques and profiling tools.

Responsabilidades

• Design scalable batch processing systems using Python and big data technologies.
• Optimize database performance, focusing on slow queries and latency improvements.
• Use Python profilers and performance monitoring tools to identify bottlenecks.
• Reduce P95 and P99 latency across the data platform.
• Build efficient ETL pipelines that handle large-scale data processing.
• Collaborate with data scientists and product teams to understand data requirements.
• Monitor and troubleshoot data pipeline issues in production.
• Implement data quality checks and validation mechanisms.
• Document data architecture and engineering processes.
• Stay current with emerging big data technologies and best practices.

Conocimientos

Python

Spark

Hadoop

SQL

NoSQL

Airflow

Profiling

ETL

Data pipelines

Educación

Bachelor's degree

Master's degree

Herramientas

Airflow

Luigi

Kubernetes

Docker

Descripción del empleo

BrightEdge, a leading SEO and content performance platform, is seeking a Big Data Engineer to design and maintain high-performance data pipelines.

You will optimize batch processing, tackle latency bottlenecks, build robust ETL workflows, and collaborate with data scientists to scale our data infrastructure.

The role requires 4+ years in data engineering, Python expertise, and hands-on Spark/Hadoop, Airflow, and SQL/NoSQL experience.