Python ETL DeveloperData Engineer Remote

ipvisibility; Ottawa; Remote; CAD 70,000 - 90,000; Full time

Job summary

A data solutions company based in Ontario is seeking an experienced professional to develop and optimize ETL jobs for data ingestion and integration. The ideal candidate will have proficiency in Python, experience with performance tuning, and the ability to handle large datasets effectively. This full-time position offers an opportunity to work with advanced data technologies in a dynamic environment.

Qualifications

• Experience in designing and developing ETL jobs.
• Proficient in Python for mapping and XML parsing.
• Ability to write efficient queries in Hive or Impala.
• Skill in performance tuning of ETL processes.

Responsibilities

• Review and develop ETL jobs to ingest data into the Data Lake.
• Parse unstructured and semi-structured data.
• Design and develop efficient mappings for data marts.
• Identify and tune performance bottlenecks in ETL jobs.

Skills

APIs

Python

AWS

MySQL

REST

Docker

Jenkins

JavaScript

Postgresql

Django

GIT

Job description

Specific Duties

• Reviewing designing developing ETL jobs to ingest data into Data Lake load data to data marts;
• extract data to integrate with various business applications.
• Parse unstructured data semi structured data such XML etc.
• Design and develop efficient Mapping and workflows to load data to Data Marts
• Map XML DTD schema in Python (customized table definitions)
• Write efficient queries and reports in Hive or Impala to extract data on ad hoc basis for data analysis.
• Identify the performance bottlenecks in ETL Jobs and tune their performance by enhancing or redesigning them.
• Responsible for performance tuning of ETL mappings and queries.
• import tables and all necessary lookup tables to facilitate the ETL process required to process daily XML files in addition to processing the very large (multi-terabytes) historical XML data files

Key Skills

APIs,Docker,Jenkins,REST,Python,AWS,NoSQL,MySQL,JavaScript,Postgresql,Django,GIT

Employment Type

Full Time

Experience

years

Vacancy

1