Aws Data Engineer

Epikso Mexico; Xico; On-site; MXN 480,000 - 650,000; Full time

Job summary

Epikso Mexico is seeking Engineers to join the On-Premises to AWS S3 Migration Platform team. You will own end-to-end migration workflows, orchestrate with AWS Step Functions, Lambda, Fargate, and Batch, and ensure data integrity through post-migration validation and reconciliation.

The role emphasizes governance, observability, and automation to reduce manual steps and improve throughput. Strong Python, shell scripting, and cloud experience are required.

Qualifications

• Education: Bachelor's or Master's in Computer Science, Applied Mathematics, Engineering, or related field.
• Experience: 3–5 years in infrastructure engineering, cloud migration, or large-scale data platform delivery.
• Languages: Strong Python (boto3, JSON processing, exception handling, logging, dataclasses, streaming files, API integration); Unix/Linux Shell Scripting.

Responsibilities

• Own end-to-end migration workflow from on-premises source buckets to AWS S3, including manifest-based and non-manifest paths.
• Build and operate AWS-based orchestration using Step Functions, Lambda, Fargate, AWS Batch, DynamoDB, SQS, and rclone.
• Implement batching, sharding, task sizing, and retry handling based on compute-tier thresholds.
• Automate pre-flight validations and enforce source freeze with deny-write policy.
• Operate rclone-based migration runtime with proper credential handling and encrypted destinations.
• Build post-migration reconciliation checks and drift detection.

Skills

Python

Shell scripting

Education

Bachelor's degree

Master's degree

Tools

rclone

AWS Lambda

Step Functions

SQS

DynamoDB

S3

KMS

Secrets Manager

Job description

Job Description:

We are seeking Engineers to join our On-Premises to AWS S3 Migration Platform team — a high-visibility program that designs, builds, and operates a secure, scalable migration platform to move object storage data from on-premises, S3-compatible object storage to AWS S3.

Responsibilities of the Engineer include:

Migration Workflow Ownership & Orchestration

1. Own the end-to-end migration workflow from on-premises source buckets to AWS S3 destination buckets, covering manifest-based and non-manifest migration paths.
2. Build and operate AWS-based orchestration using Step Functions, Lambda, Fargate, AWS Batch, DynamoDB, SQS, and rclone.
3. Implement batching, sharding, task sizing, and retry handling based on documented compute-tier thresholds (DC node, Bare Metal, manifest-required).
4. Support express-route (like-for-like) vs. exception-route migrations, using data-detectable criteria to auto-classify buckets where possible.

Execution & Validation

Automate pre-flight dry‑run validation (credentials, connectivity, readability/writability, size/object‑count limits) as a gate before any real transfer.

Implement and verify source freeze — apply deny‑write policy centrally and confirm enforcement via synthetic write‑attempt checks.

Operate the rclone‑based migration runtime with correct credential handling (Kerberos keytab, IAM role assumption via STS), KMS‑encrypted destinations, and manifest generation for large migrations.

Build and run post‑migration reconciliation checks comparing source and destination — object count/size match, per‑object checksum match, drift detection, version completeness, and configuration parity.

Track and enforce the point‑of‑no‑return cutover step — owner written acknowledgment, followed by centrally sealing the source against reads and writes.

Governance, Observability & Reporting

Implement governance for bandwidth allocation, queue management, prioritisation, throttling, and job tracking across concurrent migrations.

Design DynamoDB‑backed job, task, queue, bandwidth, and summary state tracking.

Build CloudWatch dashboards, logs, metrics, alerts, and health checks; parse migration logs into structured pass/fail evidence.

Track migration progress, escalations, and blockers via Jira; measure success on namespaces closed, not petabytes moved. Produce structured reports and dashboards for stakeholders and senior leadership.

Process Improvement & Automation

• Rigor & Follow-Through: Proactively follow up on outstanding freeze/owner sign‑offs; elevate delays per SLA with clear professional communication.
• Automation-First: Continuously convert manual verification steps (dry-run review, log inspection, reconciliation) into automated, repeatable gate checks to increase migration throughput.
• Cross‑Functional Collaboration: Work with Cloud Platform, Storage Engineering, and Networking teams to resolve connectivity and platform issues; adapt quickly to new tooling as the platform matures.

Technical Skills:

Basic Qualifications

Education: Bachelor's or Master's in Computer Science, Applied Mathematics, Engineering, or a related quantitative field.

Experience: 3–5 years of professional experience in infrastructure engineering, cloud migration, or large-scale enterprise data platform delivery.

Languages: Strong Python (boto3, JSON processing, exception handling, logging, dataclasses, streaming files, API integration); Unix/Linux Shell Scripting.

Methodology: Familiarity with change management processes in regulated financial industries.

Core Infrastructure Competencies:

1. Object Storage & S3-Compatible APIs: Deep understanding of Amazon S3, S3 Intelligent‑Tiering, S3-compatible APIs (e.g., Dell ECS or similar platforms), object metadata, prefix structures, large object transfers, versioning, and lifecycle policies.
2. Migration Tooling (rclone): Hands‑on experience configuring and operating rclone for high‑volume transfers — copy vs. sync behaviour, checksums, checkers, retries, no‑traverse behaviour, stats, logs, and bandwidth limiting.
3. AWS Serverless & Compute: Hands‑on experience with Lambda, Step Functions, SQS, SNS, EventBridge, API Gateway, SSM Parameter Store, Secrets Manager, ECS Fargate, and AWS Batch.
4. DynamoDB: Ability to design and operate tables, keys, indexes, scans, updates, batch writes, Decimal handling, and state‑management patterns for job/task tracking.
5. Networking: Understanding of AWS Direct Connect, Transit Gateway, VPCs, private subnets, S3 Gateway Endpoints, NAT Gateway scoping, and high‑throughput network design for migration traffic.
6. IAM & Security: Strong knowledge of IAM roles, cross-account access, STS assume‑role, Kerberos keytab auth, Secrets Manager, KMS encryption, and least‑privilege design.
7. Development Environments: Working knowledge of IDEs such as VS Code, IntelliJ IDEA, and PyCharm.
8. AI & Automation Tools: Familiarity with AI engines and assistants (e.g., GitHub Copilot, ChatGPT, Gemini, Claude) to accelerate scripting, validation logic, and reporting.

Core Competencies:

• Acts as a trusted team player who collaborates effectively across multiple teams and functions.
• Communicates with clarity and confidence — concise written updates, structured verbal briefings, and proactive stakeholder management.
• Works effectively with global teams across time zones and cultures; builds alignment and res eolves issues constructively.
• Delivery‑focused with a strong sense of ownership; drives work to closure and meets commitments.
• Brings high energy and urgency to achieve targets while maintaining quality and professionalism.
• Shows intellectual curiosity; asks thoughtful questions, surfaces risks early, and seeks feedback to continuously improve.