Data Engineer

Net2Source (N2S); Oregon (WI); On-site; USD 140,000 - 170,000; Full time

Job summary

Net2Source (N2S) is seeking an experienced data software engineer to design and implement data products and large-scale pipelines. You will collaborate with product owners, analysts, and business partners using Agile/Scrum practices to translate backlog items into engineering tasks and build robust distributed data processing solutions.

You will contribute to architecture, choose technologies, and ensure quality through automated tests and a mature CI/CD workflow.

Qualifications

• Bachelor’s degree in Computer Science or related technical discipline.
• 10+ years of experience in large-scale software development, 5+ years of big data experience.
• Programming experience, Python or Scala preferred.
• Experience with Hadoop and processing frameworks such as Spark, Hive.
• Experience with messaging/streaming tooling and frameworks.
• Experience with data warehousing concepts, SQL and analytical functions.
• Experience with Airflow.
• Experience with Git or Bitbucket.
• Ability to communicate effectively, verbally and in writing.
• Interest in quickly picking up new languages and technologies; Agile/Scrum experience.

Responsibilities

• Contribute to design and implement data products and features in collaboration with product owners, data analysts, and business partners using Agile / Scrum methodology.
• Contribute to overall architecture, frameworks and patterns for processing and storing large data volumes.
• Evaluate new technologies/tools/frameworks for high-volume data processing.
• Translate product backlog items into engineering tasks.
• Implement distributed data processing pipelines using big data tools.
• Build utilities and libraries to enable data flow patterns.
• Collaborate with leads to ensure quality solutions and engineering best practices.
• Build and incorporate automated tests and participate in integration testing.
• Support CI/CD frameworks and deployment processes.
• Troubleshoot data issues and perform root cause analysis.
• Work across teams to resolve operational and performance issues.

Skills

Big data

Python/Scala

Hadoop ecosystem

Spark/Hive

SQL & analytics

Agile/Scrum

Git/Bitbucket

Communication

NoSQL stores

RESTful APIs

Cloud/AWS & Databricks

Education

Bachelor’s degree in Computer Science or related field

Tools

Apache Airflow

GitHub/Bitbucket

Terraform/CloudFormation

Jenkins/CircleCI

Snowflake/Databricks

Job description

• Contribute to Design and implement data products and features in collaboration with product owners, data analysts, and business partners using Agile / Scrum methodology
• Contribute to overall architecture, frameworks and patterns for processing and storing large data volumes
• Contribute to evaluation of new technologies/tools/frameworks centered around high-volume data processing
• Translate product backlog items into logical units of work in engineering
• Implement distributed data processing pipelines using tools and languages prevalent in the big data ecosystem
• Build utilities, user defined functions, libraries, and frameworks to better enable data flow patterns
• Work with engineering leads and other teams to ensure quality solutions are implemented, and engineering best practices are defined and followed
• Build and incorporate automated unit tests and participate in integration testing efforts
• Utilize and advance continuous integration and deployment frameworks
• Troubleshoot data issues and perform root cause analysis
• Work across teams to resolve operational & performance issues

WHAT YOU BRING

• Bachelor’s degree in Computer Science, or related technical discipline
• 10+ years of experience in large-scale software development, 5+ years of big data experience
• Programming experience, Python or Scala preferred.
• Experience working with Hadoop and related processing frameworks such as Spark, Hive, etc.
• Experience with messaging/streaming/complex event processing tooling and frameworks
• Experience with data warehousing concepts, SQL and SQL Analytical functions
• Experience with workflow orchestration tools like Apache Airflow
• Experience with source code control tools like Github or Bitbucket
• Ability to communicate effectively, both verbally and written, with team members
• Interest in and ability to quickly pick up new languages, technologies, and frameworks
• Experience in Agile/Scrum application development

nice to have:

• Experience with Java
• Experience working in a public cloud environment, particularly AWS, databricks
• Experience with cloud warehouse tools like Snowflake
• Experience working with NoSQL data stores such as HBase, DynamoDB, etc.
• Experience building RESTful API’s to enable data consumption
• Experience with build tools such as Terraform or CloudFormation and automation tools such as Jenkins or Circle CI
• Experience with practices like Continuous Development, Continuous Integration (CI/CD) and Automated Testing