Data Engineer

Infoverity; Dublin (OH); Remote; USD 80,000 - 110,000; Full time

Job summary

Infoverity is looking for an experienced Data Engineer to join their consulting team. In this fully remote role, you'll design cloud-based data solutions, collaborate with scientists and analysts, and optimize data architectures to support business needs. The position offers a chance to impact clients' data strategies heavily while working with cutting-edge technologies like Databricks.

Qualifications

• Minimum of 2 to 3 years of experience in data engineering and architecture.
• Strong expertise in Databricks and cloud platforms.
• Proficiency in SQL, Python, and distributed data processing frameworks.

Responsibilities

• Design and implement scalable, cloud-based data pipelines.
• Develop and optimize ETL/ELT workflows and data warehousing.
• Work with clients to understand needs and deliver data solutions.

Skills

Data Engineering

Cloud Technologies

SQL

Python

Spark

AI/ML Workflows

Data Governance

Communication

Client-facing Skills

Tools

Databricks

Snowflake

Microsoft Fabric

ETL/ELT

Python

Apache Airflow

Job description

This is a remote position.

Infoverity is seeking an experienced Data Engineer to join our dynamic consulting team, helping clients architect, build, and optimize modern data platforms. In this role, you will design and implement scalable data solutions, leveraging cloud technologies, specifically Databricks, to drive innovation in AI/ML, data warehousing, and data integration. As a key player at Infoverity you will collaborate with cross-functional teams, including data scientists, business analysts, and solution architects, to deliver high-impact data solutions tailored to our clients’ needs.

Key Responsibilities

• Data Engineering & Architecture: Design and implement scalable, cloud-based data pipelines to ingest, transform, and store data from various sources.
  

• Data Warehousing & Integration: Develop and optimize ETL/ELT workflows, ensuring efficient data movement between systems, specifically Databricks.
  

• AI/ML Enablement: Work alongside data science teams to support feature engineering, model training, and ML operations (MLOps) in cloud environments.
  

• Data Modeling & Governance: Develop data models, schemas, and best practices to ensure data integrity, consistency, and security.
  

• Performance Optimization: Monitor, troubleshoot, and optimize query performance, ensuring scalability and efficiency.
  

• Consulting & Client Engagement: Work directly with clients to understand business needs, recommend best practices, and deliver tailored data solutions.
  

• Data Architecture Design: Design and implement scalable, high-performance data architectures using Databricks.
  

• Requirement Gathering: Lead requirement-gathering sessions with clients and internal teams to understand business needs and define best practices for solutions including data workflows.
  

• Cloud Collaboration: Collaborate with cloud platform teams to optimize data storage and retrieval in environments like AWS S3, Azure Data Lake, and Delta Lake, among others.
  

• Workflow Optimization: Translate complex data processes, such as those in Alteryx and Tableau, into optimized Databricks workflows using PySpark and SQL.
  

• Automation Development: Develop reusable automation scripts to streamline workflow migrations and improve operational efficiency.
  

• Development Support: Provide hands-on development and troubleshooting support to ensure smooth implementation and optimal performance.
  

• Governance & Best Practices: Partner with cross-functional teams to establish data governance frameworks, best practices, and standardized reporting processes.
  

• Training & Support: Deliver training, documentation, and ongoing support to empower users and enhance organizational data literacy.
  

Requirements

Required Qualifications

• Minimum of 2 to 3 years of experience in data engineering, data architecture, and data integration.
  

• Strong expertise in Databricks, Snowflake, and/or Microsoft Fabric.
  

• Proficiency in SQL, Python, Spark, and distributed data processing frameworks.
  

• Experience with cloud platforms (Azure, AWS, or GCP) and their native data services.
  

• Hands-on experience with ETL/ELT development, data pipelines, and data warehousing.
  

• Knowledge of AI/ML workflows, including feature engineering and ML model deployment.
  

• Strong understanding of data governance, security, and compliance best practices.
  

• Excellent problem-solving, communication, and client-facing consulting skills.
  

• Ability to work independently and as part of a team.
  

Preferred Qualifications

• Certifications in Databricks, Snowflake, Microsoft Fabric, or a cloud platform (Azure, AWS, GCP).
  

• Experience with Apache Airflow, dbt, Delta Lake, or similar data orchestration tools.
  

• Familiarity with DevOps, CI/CD pipelines, and Infrastructure as Code (IaC) tools like Terraform.