Remote Senior Data Engineer — ETL & Data Quality Expert

CGI Group, Inc.; Portland (OR); Remote; USD 63,000 - 139,000; Full time

Benefits offered by this job

Profit participation program

Medical, dental, and vision benefits

401(k) matching and CGI share purchase

Job summary

CGI Group, Inc. is hiring a Senior Data Engineer to design and execute end-to-end test plans for enterprise data platforms and ETL/ELT pipelines.

You will validate data accuracy, transformations and quality across source and target systems, writing complex SQL and leveraging Databricks, Spark, PySpark and Python. The role requires hands-on testing experience with Hadoop/HDFS, Hive, and Azure Data Factory, in a Unix/Linux environment, and collaboration within Agile/Scrum teams.

Qualifications

• 8+ years of experience in software quality engineering/testing with emphasis on ETL testing, data validation, integration testing and regression testing.
• Strong SQL skills with complex queries for source-to-target validation, reconciliation and data quality testing.
• Experience testing large-scale data platforms and ETL/ELT pipelines.
• Hands-on with Databricks, Apache Spark, PySpark, Python and Azure Data Factory (ADF).
• Experience with Hadoop ecosystem technologies including Hadoop/HDFS and Hive.
• Experience working in Unix/Linux and using enterprise job scheduling tools such as Control M.
• Solid understanding of data quality concepts, data transformations, data mappings, and source-to-target validation.
• Experience with defect management and test management processes in Agile/Scrum delivery environments.
• Strong analytical, troubleshooting, communication and collaboration skills.

Responsibilities

• Design, develop, and execute test plans, scenarios, and cases for enterprise data platforms, ETL/ELT pipelines and data integration solutions.
• Run end-to-end ETL testing and validate data accuracy, completeness, transformation logic, reconciliation and data quality across systems.
• Write complex SQL queries to support backend testing and data validation.
• Test large-scale data processing solutions using Databricks, Apache Spark, PySpark, Python, Hive, Hadoop/HDFS and Azure Data Factory (ADF).
• Perform functional, integration, regression, system and data quality testing for data pipelines and data-driven applications.
• Validate batch processing, job scheduling, dependencies and workflow execution using Control M and Unix/Linux environments.
• Identify, document, track and retest defects while partnering with developers, data engineers and business analysts.
• Develop and maintain reusable test scripts and automation where applicable.
• Participate in Agile/Scrum activities including sprint planning, refinement, daily stand ups, reviews and retrospectives.

Skills

ETL testing

Data validation

Integration testing

Regression testing

SQL

Databricks

Apache Spark

PySpark

Python

Azure Data Factory

Hadoop/HDFS

Hive

Unix/Linux

Control-M

Data quality

Defect management

Agile/Scrum

Analytical thinking

Tools

Databricks

Apache Spark

PySpark

Python

Azure Data Factory

Hadoop/HDFS

Hive

Unix/Linux

Control-M

Job description

CGI Group, Inc. is hiring a Senior Data Engineer to design and execute end-to-end test plans for enterprise data platforms and ETL/ELT pipelines.

You will validate data accuracy, transformations and quality across source and target systems, writing complex SQL and leveraging Databricks, Spark, PySpark and Python. The role requires hands-on testing experience with Hadoop/HDFS, Hive, and Azure Data Factory, in a Unix/Linux environment, and collaboration within Agile/Scrum teams.