Remote Data Engineer: Scalable Pipelines & ML Features

Eliassen Group; Dover (DE); A distancia; USD 83.000 - 96.000; A tiempo parcial

Descripción de la vacante

Eliassen Group is seeking a Data Engineer to join remotely on a contract basis. The role focuses on building and optimizing scalable data pipelines across Azure and GCP, implementing feature engineering for machine learning with Python and Vertex AI, and ensuring quality through automated testing and CI/CD.

You will modernize data assets across SQL, NoSQL, and streaming platforms, document architectures, and own systems and interfaces across the SDLC while collaborating with cross-functional

Formación

• 4+ years of professional data development experience.
• 4+ years with SQL and NoSQL technologies.
• 3+ years building and maintaining data pipelines and workflows.
• 2+ years developing with Python.
• Experience with feature engineering for machine learning pipelines.
• Experience with CI/CD pipelines and processes.
• Experience with automated unit, integration, and performance testing.
• Experience with version control such as Git.
• Understanding of ETL and data warehousing concepts.
• Understanding of Agile principles, including Scrum.
• Knowledge of structured streaming with Spark, Kafka, or EventHub (preferred).
• Experience with GCP services such as BigQuery, Vertex AI Platform, Cloud Storage, AutoMLOps, and Dataflow, or Databricks equivalents (preferred).
• Experience with GitHub SaaS and GitHub Actions (preferred).
• Understanding of Databricks concepts, PySpark, and Spark development (preferred).
• Experience with service-oriented architecture (preferred).

Responsabilidades

• Design, build, and maintain scalable and reliable data pipelines for ingestion, transformation, and integration using tools such as Kafka and Databricks.
• Leverage innovative technologies to modernize and extend core data assets across SQL, NoSQL, cloud, and real-time streaming platforms.
• Develop and manage feature engineering pipelines for machine learning using Vertex AI, BigQuery ML, and Python.
• Design and implement automated unit, integration, and performance testing frameworks to ensure data quality and reliability.
• Optimize data workflows for performance, cost efficiency, and scalability across large datasets and complex environments.
• Draft and review architectural diagrams, interface specifications, and design documents to communicate data solutions and requirements.
• Own systems, processes, and tech stack elements and drive features to completion across internal and external applications.

Conocimientos

SQL & NoSQL

Python

CI/CD

Git

Feature engineering

Data pipelines

Testing frameworks

ETL concepts

Agile/Scrum

Spark / Kafka

GCP / Databricks

GitHub Actions

PySpark

SOA

Herramientas

Kafka

Databricks

Vertex AI

BigQuery

GitHub Actions

Descripción del empleo

Eliassen Group is seeking a Data Engineer to join remotely on a contract basis. The role focuses on building and optimizing scalable data pipelines across Azure and GCP, implementing feature engineering for machine learning with Python and Vertex AI, and ensuring quality through automated testing and CI/CD.

You will modernize data assets across SQL, NoSQL, and streaming platforms, document architectures, and own systems and interfaces across the SDLC while collaborating with cross-functional