Data Engineer II: Pipelines, Governance & API Connectors

Bow Wave LLC; Reston (VA); On-site; USD 120,000 - 160,000; Full time

Job summary

Bow Wave LLC is seeking a data engineering professional to design and optimize data pipelines using Spark, Hudi and EMR, while managing data pedigree and security. You will clean and preprocess data for analytics, collaborate with enterprise groups to elevate data standards, and architect scalable ETL processes with engineers, stewards, and partners.

API connector development and database administration are also key responsibilities.

Responsibilities

• Design and optimize Data Pipelines leveraging Spark, Hudi, EMR cloud services and Kubernetes containers
• Maintain pedigree and provenance of data to protect access
• Clean and preprocess data for analytic use
• Collaborate with enterprise groups to advance data standards
• Work with engineering, data stewards, and partners to create repeatable ETL processes
• Provide advanced database administration support across Oracle, MySQL, MariaDB, MongoDB, Elastic
• Develop API connectors for ingesting new catalog entries from databases and files
• Ensure data mappings deliver optimal performance for user experience
• Support data governance and security requirements across data platforms

Skills

Analytical thinking

Collaborative

Communication

Tools

Spark

Hudi

EMR

Kubernetes

Oracle

MySQL

MariaDB

MongoDB

Elastic

Job description

Bow Wave LLC is seeking a data engineering professional to design and optimize data pipelines using Spark, Hudi and EMR, while managing data pedigree and security. You will clean and preprocess data for analytics, collaborate with enterprise groups to elevate data standards, and architect scalable ETL processes with engineers, stewards, and partners.

API connector development and database administration are also key responsibilities.