data engineer vehicle telemetry

HireHi; United States; Remote; USD 120,000 - 180,000; Full time

Job summary

HireHi ищет опытного инженера по данным для проекта по сбору и анализу телеметрии транспортных средств в рамках глобальной платформы на Azure Databricks. Платформа работает на масштабe по регионам Европа, Северная Америка и Китай, с удаленной работой.

В кандидате приветствуются 3+ года опыта, глубокие знания Azure Databricks, PySpark и Python, умение работать с мониторингом, observability и оптимизацией затрат в облаке. Английский не ниже B2.

Qualifications

• 3+ года hands-on опыта в data engineering.
• Опыт работы с Azure Databricks и PySpark.
• Умение разбирать и оптимизировать большие Spark-работы.

Responsibilities

• Мониторинг состояния, стабильности и производительности production data platform.
• Расследование проблем производительности и инцидентов.
• Анализ нагрузок Databricks и поиск бутылочных мест.
• Оптимизация PySpark и Databricks задач по производительности, надежности и затратам.
• Обслуживание примерно 200 существующих Databricks задач и пайплайнов.
• Подготовка платформы к росту объёмов данных и рабочих нагрузок.
• Разработка масштабируемых стратегий с фокусом на предсказуемые инфраструктурные расходы.
• Мониторинг потребления Databricks и Azure ресурсов.

Skills

Azure Databricks

PySpark

Python

Distributed data processing

Performance optimization

Cloud cost optimization

Monitoring & observability

GitHub Actions

English (B2+)

Tools

GitHub Actions

Job description

Описание:

The project collects, processes, and analyzes vehicle telemetry data for a major European automotive manufacturer. Its Azure Databricks platform operates at significant scale across multiple regions, including Europe, North America, and China.

Задачи:

• Monitor the health, stability, and performance of the production data platform;
• Investigate and resolve performance issues and production incidents;
• Analyze Databricks workloads and identify performance bottlenecks;
• Optimize PySpark workloads and Databricks jobs for performance, reliability, and cost efficiency;
• Support and improve approximately 200 existing Databricks jobs and associated data pipelines;
• Prepare the platform for increasing data volumes and workloads as additional vehicle platforms are onboarded;
• Design and implement scaling strategies with a strong focus on predictable infrastructure and operational costs;
• Track and optimize Databricks and Azure resource consumption;
• Improve platform observability, monitoring, alerting, and operational processes;
• Ensure consistency across multiple regional deployments in Europe, North America, and China;
• Identify opportunities for technical improvements, automation, and reduction of operational overhead.

Требования:

• 3+ Years of hands-on experience in data engineering;
• Expertise in Azure Databricks;
• Proficiency in PySpark and distributed data processing;
• Advanced Python development skills;
• Understanding of performance optimization for large-scale Spark workloads;
• Experience operating and troubleshooting production data platforms;
• Knowledge of monitoring, observability, and capacity planning combined with cloud cost optimization;
• Ability to analyze existing systems, identify bottlenecks, and propose pragmatic improvements;
• English proficiency at an Upper-Intermediate level (B2) or higher;
• Nice to have: familiarity with Microsoft Azure services and cloud infrastructure, experience building and maintaining CI/CD pipelines using GitHub Actions, exposure to large-scale telemetry, IoT, or automotive data, experience managing Databricks environments with a large number of scheduled jobs and pipelines, multi-region or geographically distributed cloud deployments.

Условия:

Remote work is available in Georgia and four other locations.