Data Engineer

Stellent IT LLC; Wilmington (DE); Ibrido; USD 83.000 - 152.000; Tempo pieno

Descrizione del lavoro

Stellent IT LLC seeks a hands-on Data Engineer to support a high-priority banking migration project in Wilmington, DE. You will design, build, and operate scalable data pipelines using Java, Spark, PySpark, and AWS services, with a focus on data quality and production readiness.

You'll collaborate across teams to deliver ingestion, transformation, and validation workflows, implement CI/CD, and ensure robust monitoring and support for data migrations in a large enterprise environment.

Competenze

• Strong experience in data engineering and data pipeline development.
• Hands-on Java development and Spring Boot experience.
• Proficient with Apache Spark and PySpark.
• Experience with AWS Glue, Lambda, and Amazon S3.
• Familiar with CI/CD pipelines and automated software delivery.
• Experience in large-scale enterprise environments and data migration.

Mansioni

• Design, develop, and maintain scalable batch and streaming data pipelines.
• Build data ingestion, transformation, validation, and delivery processes.
• Develop production-quality applications using Java and Spring Boot.
• Use Spark and PySpark to process large datasets.
• Develop AWS-based data solutions with Glue, Lambda, and S3.
• Create automated tests and participate in code reviews.
• Build and maintain CI/CD pipelines for deployment.
• Monitor pipelines and troubleshoot production issues.
• Support data migration and integration with APIs and databases.
• Ensure data quality checks and error handling.

Conoscenze

Data Engineering

Java development

Spring Boot

Apache Spark

PySpark

AWS Glue

AWS Lambda

Amazon S3

CI/CD

Data pipelines

Production support

Data ingestion

Data transformation

Data validation

Troubleshooting

Strumenti

Kafka

Docker

Kubernetes

REST APIs

Microservices

Descrizione del lavoro

Job title: Data Engineer

Contract Duration: 4 months, Contract W2 Only Potential Extension

Primary Locations: Wilmington, DE Hybrid

Other Locations Considered: McLean, VA; Richmond, VA; New York City, NY; Plano, TX

Please submit the candidate who can take the assessment today for this particular position.

Position Overview

Client is seeking experienced Data Engineers to support a high-priority customer migration initiative involving the transition of Discover customers onto the platform. The selected candidate will design, develop, maintain, and support scalable data pipelines and data-processing solutions in a high-volume banking environment. This is primarily a Java-focused Data Engineering role. Candidates must have strong hands-on experience with Java, Spring Boot, Apache Spark, PySpark, AWS, and CI/CD.

The ideal candidate will have experience across data ingestion, transformation, integration, testing, deployment, monitoring, and production support.

Key Responsibilities

• Design, develop, and maintain scalable batch and streaming data pipelines.
• Build data ingestion, transformation, validation, and delivery processes.
• Develop production-quality applications and services using Java and Spring Boot.
• Use Apache Spark and PySpark to process and transform large datasets.
• Develop AWS-based data solutions using AWS Glue, AWS Lambda, and Amazon S3.
• Build reusable frameworks and components to improve pipeline performance and reliability.
• Implement data quality checks, validation, reconciliation, and error-handling processes.
• Support customer and account data migration while ensuring data accuracy and completeness.
• Integrate data pipelines with APIs, databases, messaging systems, and downstream platforms.
• Develop RESTful APIs and supporting services when required.
• Create automated unit, integration, and functional tests.
• Participate in code reviews, technical design discussions, and documentation.
• Build and maintain CI/CD pipelines for automated build, testing, and deployment.
• Monitor pipeline execution and troubleshoot production failures.
• Perform root-cause analysis and resolve production issues.
• Optimize Spark jobs, data-processing workflows, and cloud resources.
• Collaborate with architects, developers, QA, DevOps, product managers, and business stakeholders.
• Support deployments across development, testing, staging, and production environments.
• Follow Client's security, data governance, risk-management, and software-development standards.

Required Qualifications / Must-Have Skills

• Strong professional experience in Data Engineering and Data Pipeline Development.
• Strong hands-on Java development experience.
• Hands-on experience with Spring Boot.
• Strong experience with Apache Spark.
• Strong experience with PySpark.
• Hands-on experience with AWS Glue.
• Experience with AWS Lambda.
• Strong experience with Amazon S3 and data lake solutions.
• Experience designing, developing, testing, and supporting scalable data pipelines.
• Experience with CI/CD pipelines and automated software delivery.
• Strong understanding of data ingestion, transformation, validation, reconciliation, and error handling.
• Production troubleshooting and support experience.
• Experience working in large-scale enterprise environments.
• Strong communication and problem-solving skills.

Preferred / Nice-to-Have Skills

• Python development experience.
• RESTful API and microservices development.
• Kafka or other event-streaming technologies.
• Customer data migration experience.
• Banking, financial services, payments, credit card, or related domain experience.
• AI/ML or AI-enabled data platform exposure.
• Cloud-native application development.
• Distributed systems experience.
• Data governance, security, privacy, and regulatory experience.
• Infrastructure automation.
• Docker/Kubernetes experience.

Technology Stack

Core:

• Java
• Spring Boot
• Apache Spark
• PySpark
• AWS Glue
• AWS Lambda
• Amazon S3
• CI/CD

Additional:

• Python
• REST APIs
• Microservices
• Kafka
• Kubernetes
• Cloud-native technologies

Ideal Candidate Profile

The ideal candidate is a hands-on Data Engineer with strong Java development experience who has built and supported enterprise-scale data pipelines. The candidate should be comfortable working across Java/Spring Boot, Spark/PySpark, AWS Glue, Lambda, S3, CI/CD, data migration, and production support.

Candidates with banking/financial services or customer data migration experience, particularly previous Client experience, would be highly relevant.

Screening Must Have

1. Strong Data Engineering experience.
2. Strong hands-on Java development.
3. Spring Boot experience.
4. Apache Spark experience.
5. PySpark experience.
6. AWS Glue experience.
7. AWS Lambda experience.
8. Amazon S3 experience.
9. Data pipeline development and production support.
10. CI/CD experience.
11. Data ingestion, transformation, validation, and reconciliation experience.
12. Strong troubleshooting and problem-solving skills.