Senior Data Engineer – Real-Time Pipelines & Data Solutions

American Express; Hoschton (GA); Remote; USD 57,859,200 - 71,635,200; Full time

Job summary

American Express is seeking a Senior Data Engineer to own design, development, testing and deployment of high‑performance data pipelines and analytics solutions for HR data services. You will collaborate with product, analytics and data engineering colleagues to accelerate digital adoption, experimentation and customer insights across platforms.

The role focuses on data architecture, scalable workflows, CI/CD deployments and governance, with cross‑functional collaboration across teams.

Qualifications

• At least 5 years of software development experience in big-data technologies.
• Proficiency with Python, Spark (PySpark, Spark SQL), Hive, and Shell scripting.
• Hands-on experience building data ingestion pipelines and data consumption services (e.g., data platforms).
• Strong knowledge of large-scale data processing frameworks such as Spark, MapReduce, YARN, Hive.
• Experience with NoSQL databases (HBase, Cassandra, Redis, Memcached).
• Familiarity with cloud platforms (GCP preferred) and distributed data services.
• Experience with CI/CD pipelines using GitHub, Jenkins and XLR.

Responsibilities

• Define data architecture, design and integration solutions for enterprise data services.
• Write, test, deploy and maintain scalable data engineering workflows including batch, stream and real-time pipelines.
• Lead and mentor an agile engineering team through code reviews, CI/CD, documentation and best practices.
• Debug, troubleshoot and optimize data application performance.
• Automate deployments to development, QA and production environments using CI/CD pipelines.
• Implement data governance, security and compliance controls to meet regulatory requirements.
• Collaborate with analytics, product and data scientists to build dashboards and BI solutions.
• Participate in coding standards and technical design reviews for data services.

Skills

Python

Spark PySpark

Hive

Shell scripting

Big-data

Data ingestion pipelines

CI/CD

Distributed data processing

NoSQL databases

Cloud platforms

Spark streaming

MapReduce

YARN

HBase

Cassandra

Redis

Memcached

Kubernetes

Tools

GitHub

Jenkins

XLR

Job description

American Express is seeking a Senior Data Engineer to own design, development, testing and deployment of high‑performance data pipelines and analytics solutions for HR data services. You will collaborate with product, analytics and data engineering colleagues to accelerate digital adoption, experimentation and customer insights across platforms.

The role focuses on data architecture, scalable workflows, CI/CD deployments and governance, with cross‑functional collaboration across teams.