Data & Infrastructure Engineer

Value Innovation Labs Inc.; United States; On-site; USD 140,000 - 200,000; Full time

Job summary

Value Innovation Labs Inc. in the United States seeks a Senior Data and Infrastructure Engineer to design, deploy, and optimize enterprise-scale data pipelines and compute resources.

You will lead provisioning of VMs and Kubernetes clusters, collaborate with data science teams, define repository structure, and address governance and reconciliation across data sources. The role emphasizes software development best practices, documentation, and scalable architectures for large datasets, including

Qualifications

• 8+ years of data engineering experience delivering complex projects.
• Expert in ELT/ETL and data warehousing.
• Strong Python for data processing.

Responsibilities

• Provision, create, and operate VMs and Kubernetes clusters for load-balanced data processing for large data sets (~10 TB)
• Collaborate with business and data science personnel to define strategy and approach for data pipeline testing
• Define organization structure for code repository
• Troubleshoot complex data-related issues including data reconciliation across sources and governance.

Skills

Data Engineering

Python

SQL

ELT/ETL

Data Warehousing

Kubernetes

VM provisioning

Git

NET ecosystem

Tools

Git

Kubernetes

VMs

Data orchestration

Job description

• Provision, create, and operate VMs and Kubernetes clusters for load-balanced data processing for large data sets (~10 TB)
• Collaborate with business and data science personnel to define strategy and approach for data pipeline testing
• Define organization structure for code repository
• Troubleshoot complex data-related issues including data reconciliation across sources and governance.

Job Description

The Senior Data and Infrastructure Engineer provides expert-level knowledge in data integration techniques and computation infrastructure to design, build, and deploy computing resources (VMs, Kubernetes clusters, local Git environments) utilizing on-premises devices and data ingestion pipelines. This role's focus is applying best practices of software development to the design, implementation, automation, and optimization of data pipelines and enterprise-wide data-centric applications (e.g., data lakes, data warehouses) for large and/or challenging data sets. This role will provide guidance, oversight, and technical thought leadership in both the computational infrastructure and data pipeline engineering spaces, including ensuring that all computational and data resources are well documented.

Qualifications

• 8+ years of experience in Data Engineering and in delivering complex projects
• Expert-level data engineering knowledge including ELT/ETL, Data Warehousing
• Strong Python experience for data processing
• Experience with computational infrastructure provisioning (VMs and Kubernetes clusters) and deployment, preferably within ecosystems operating on .NET frameworks.
• Strong SQL writing and optimizing skills for relational database platforms
• Extensive experience with managing pipelines that process large volumes of data
• Extensive experience with workflow orchestration/scheduling solutions