Data Engineer 2 - Applicants must be US Citizens with TS - FSP to apply

Bow Wave LLC; Reston (VA); On-site; USD 120,000 - 160,000; Full time

Job summary

Bow Wave LLC is seeking a data engineering professional to design and optimize data pipelines using Spark, Hudi and EMR, while managing data pedigree and security. You will clean and preprocess data for analytics, collaborate with enterprise groups to elevate data standards, and architect scalable ETL processes with engineers, stewards, and partners.

API connector development and database administration are also key responsibilities.

Responsibilities

• Design and optimize Data Pipelines leveraging Spark, Hudi, EMR cloud services and Kubernetes containers
• Maintain pedigree and provenance of data to protect access
• Clean and preprocess data for analytic use
• Collaborate with enterprise groups to advance data standards
• Work with engineering, data stewards, and partners to create repeatable ETL processes
• Provide advanced database administration support across Oracle, MySQL, MariaDB, MongoDB, Elastic
• Develop API connectors for ingesting new catalog entries from databases and files
• Ensure data mappings deliver optimal performance for user experience
• Support data governance and security requirements across data platforms

Skills

Analytical thinking

Collaborative

Communication

Tools

Spark

Hudi

EMR

Kubernetes

Oracle

MySQL

MariaDB

MongoDB

Elastic

Job description

Requirements

• Design and optimize Data Pipelines using Spark, Hudi, EMR cloud services, and Kubernetes containers
• Make sure pedigree and provenance of the data is maintained such that the access to data is protected
• Clean and preprocess data to enable analytic access
• Collaborate with enterprise working groups to advance the state of data standards
• Collaborate with the engineering team, data stewards, and mission partners to aid in getting actionable value out of the data holdings architects complex, repeatable ETL processes
• Provide Advanced Database Administration support in Oracle, MySQL, MariaDB, MongoDB, Elastic and others
• Supports Experience with Targeting using Sponsor Tools, Reverse Engineering
• Develop API connectors to enable ingest of new data catalog entries from databases and files
• Ensure that data mappings will provide the best performance for expected user experience