Datastage/ Python developer (1148872)

The Judge Group; Columbus (OH); On-site; USD 90,000 - 130,000; Full time

Job summary

The Judge Group in Columbus, OH is seeking a Data Engineer to design and implement ETL pipelines, data integration, and data warehousing solutions across on-prem and cloud environments.

You will work with DataStage, Hadoop, Spark, Kafka, and Snowflake to automate data flows, ensure data quality, and support analytics initiatives for business users. Collaboration with analysts and governance alignment are key parts of the role.

Qualifications

• Bachelor's degree in computer science or related field.
• Experience with ETL and data integration in enterprise environments.
• Knowledge of big data technologies and data warehousing concepts.

Responsibilities

• Develop shell scripts for automation of data flow and ETL orchestration.
• Implement ETL transformations using DataStage and Hadoop ecosystems.
• Create one-time/data extractions from data marts for analytics.
• Migrate data to data lake and build Hadoop data lake.
• Design on-prem or cloud data warehousing models using ETL tools and cloud platforms.
• Coordinate with business users to understand data sources and entities.
• Prepare technical design documents and data models.
• Perform testing and ensure CI/CD alignment.

Skills

ETL development

Data engineering

Big data

Data warehousing

Education

Bachelor's degree in Computer Science or related field

Tools

DataStage

Hadoop

Spark

Hive

Pig

Sqoop

Kafka

AWS

Snowflake

Oracle

Python

Infosphere DataStage

Job description

Location

Columbus, OH

Salary

Depends on Experience

Job Description

• Develop Shell scripts for automation of the data flow process and ETL orchestration pipeline.
• Implement ETL transformations using DataStage, Hadoop Big data technologies like Spark-Scala/Spark-Java, Hive, or Pig based on the business use case.
• Develop One-time/On-demand data Extractions from the data marts in the database systems for the business users for analytical purposes.
• Direct Podium for end-to-end data management.
• Develop automation and ETL jobs to schedule data processing utilities using reusable ETL components and frameworks of the data pipelines.
• Implement Data profiling, Data quality, and Data standardization for all the workflows based on EDM standards (Enterprise Data Management).
• Build Hadoop Data Lake for analytics using Hadoop echo system.
• Migrate tables from traditional RDBMS into Data Lake using Sqoop.
• Transfer real-time data from different data sources into HDFS systems using Kafka producers, consumers, and Kafka brokers.
• Use cloud-native development, micro-services architecture, operations, and governance framework to gather required technical data for purposes of formulating recommended customization of AWS.
• Perform Unit and regression testing, prepare unit test case documents for comparing the results with business requirements.
• Ensure the technical implementations are in line with the requirements and specifications.
• Help Quality Analyst Team in preparing a comprehensive test plan to ensure proper functioning of the application for the zero-defect delivery for the continuous integration and deployment to the production.
• Prepare Technical Specification documents that explain the technical solutions and list the technologies to be used while implementing the solution.
• Prepare data models to be certain that the data objects are represented accurately by the ETL and Data engineering software and application tools.
• Participate in business requirement meetings, perform requirement, and impact analysis with Business Analysts.
• Coordinate with business users to understand the data sources/entities.
• Design the on-prem or cloud data warehousing models using ETL tools such as IBM Infosphere Data Stage, Oracle, Python, and cloud platforms such as Snowflake and AWS (Amazon Web Services)
• Create Technical design documents as part of project deliverables.
• Analyse Teradata databases to design extract strategies into Big data and cloud platforms.
• These duties are complex and require at least a bachelor’s degree in computer science or a related field.

By providing your phone number, you consent to: (1) receive automated text messages and calls from the Judge Group, Inc. and its affiliates (collectively \"Judge\") to such phone number regarding job opportunities, your job application, and for other related purposes. Message & data rates apply and message frequency may vary. Consistent with Judge's Privacy Policy, information obtained from your consent will not be shared with third parties for marketing/promotional purposes. Reply STOP to opt out of receiving telephone calls and text messages from Judge and HELP for help.

Contact: [e-mail]

This job and many more are available through The Judge Group. Find us on the web at www.judge.com