Hybrid Data Engineer: Ab Initio & Spark Expert

Motion Recruitment Partners LLC; Charlotte (NC); Hybrid; USD 120,000 - 150,000; Full time

Job summary

Motion Recruitment Partners LLC is seeking a Software Engineer for a long-term contract in Charlotte, NC. The role is hybrid, offering exposure to a leading financial services institution and modern data pipelines.

You will build and maintain scalable data processing workflows, collaborate with product and analytics teams, and leverage cloud and on-prem technologies to deliver reliable, compliant solutions.

Qualifications

• 4+ years of Data Engineering experience or equivalent shown through work experience, training, military experience, education.
• 4+ years PL/SQL and SQL skills with proven experience in Oracle, Teradata, Python and/or BigQuery: complex query development, tuning, and debugging.
• 3+ years programming skills in Python; hands-on PySpark for distributed data processing.
• 3+ years of ETL/ETL design, data warehousing concepts, and data modeling best practices.

Responsibilities

• Build and maintain scalable batch and near real-time data pipelines using AB Initio, Python, PySpark, PL/SQL and SQL to ingest, transform, and publish curated datasets across on-prem and Google Cloud platforms.
• Develop and optimize BigQuery transformations and datamodels, including partitioning, clustering, query optimization, and cost/performance tuning.
• Support modernization/migration from Teradata and Ab Initio workflows to GCP/BigQuery, including logic re-platforming, reconciliation, parallel runs, and controlled cutovers.
• Implement orchestration and scheduling for pipelines using legacy Autosys while driving migration toward Google Cloud Composer (Airflow), including dependency management, retries, SLAs, and backfills.
• Apply data governance and discovery practices using Dataplex: metadata management, dataset organization, classification support, and ensuring data is consumption-ready.
• Build and operationalize data quality controls using Informatica Data Quality: profiling, rule implementation, thresholds, exception handling, and embedding quality gates into pipelines.
• Ensure operational excellence: monitoring, alerting, runbooks, incident triage/root cause analysis, and continuous improvements to reliability and performance.
• Implement secure data engineering practices: least-privilege access, PII handling/masking where required, retention controls, and audit-friendly documentation.
• Partner with product, analytics, and engineering stakeholders to translate requirements into clear data contracts, curated datasets, and maintainable documentation.
• Must-have: Use AI-assisted coding tools to accelerate development while maintaining strong code review discipline, testing, and secure coding standards.

Skills

Data Engineering

PL/SQL

SQL

Python

BigQuery

Ab Initio

PySpark

ETL

Data warehousing

GCP

Airflow

Autosys

CI/CD tooling

AI-assisted coding

Dataplex

Job description

Motion Recruitment Partners LLC is seeking a Software Engineer for a long-term contract in Charlotte, NC. The role is hybrid, offering exposure to a leading financial services institution and modern data pipelines.

You will build and maintain scalable data processing workflows, collaborate with product and analytics teams, and leverage cloud and on-prem technologies to deliver reliable, compliant solutions.