Remote Data Engineer for Scalable AI Pipelines

Eliassen Group; Richmond (VA); Remote; USD 83,000 - 96,000; Full time

Benefits offered by this job

Medical benefits

Dental benefits

Vision benefits

401k with match

Life insurance

Job summary

Eliassen Group is seeking a Data Engineer contractor to build and optimize scalable data pipelines across Azure and GCP, leveraging Kafka and Databricks for ingestion, transformation, and real-time analytics.

You will develop feature engineering pipelines for ML using Vertex AI and BigQuery ML, ensure quality with automated testing and CI/CD, and document architectures and interfaces across SQL, NoSQL, and streaming platforms.

Qualifications

• 4+ years of professional data development experience.
• 4+ years with SQL and NoSQL technologies.
• 3+ years building and maintaining data pipelines and workflows.
• 2+ years developing with Python.
• Experience with feature engineering for machine learning pipelines.
• Experience with CI/CD pipelines and processes.
• Experience with automated unit, integration, and performance testing.
• Experience with version control such as Git.
• Understanding of ETL and data warehousing concepts.
• Understanding of Agile principles, including Scrum.

Responsibilities

• Design, build, and maintain scalable data pipelines for ingestion, transformation, and integration using Kafka and Databricks.
• Modernize data assets across SQL, NoSQL, cloud, and real-time streaming platforms.
• Develop feature engineering pipelines for ML using Vertex AI and BigQuery ML.
• Implement automated testing frameworks to ensure data quality and reliability.
• Optimize data workflows for performance, cost, and scalability.
• Draft architectural diagrams and interface specifications for data solutions.

Skills

SQL

NoSQL

Python

CI/CD

Git

ETL

Agile/Scrum

Kafka

Databricks

BigQuery

Vertex AI

Spark

GitHub Actions

Tools

Azure

GCP

Databricks

Vertex AI

BigQuery

Kafka

Spark

GitHub Actions

Job description

Eliassen Group is seeking a Data Engineer contractor to build and optimize scalable data pipelines across Azure and GCP, leveraging Kafka and Databricks for ingestion, transformation, and real-time analytics.

You will develop feature engineering pipelines for ML using Vertex AI and BigQuery ML, ensure quality with automated testing and CI/CD, and document architectures and interfaces across SQL, NoSQL, and streaming platforms.