Senior Data Engineer: AI-Driven Pipelines on AWS

iScale Solutions; Makati; On-site; PHP 1,200,000 - 1,800,000; Full time

Job summary

iScale Solutions in Makati, Philippines, is seeking an experienced data engineer to own end-to-end data pipelines, enabling reliable ingestion and processing of large-scale datasets across platforms.

You will design scalable architectures, implement robust QA, and leverage AWS services, Hive, Spark, and Airflow while collaborating with analysts, product, and data science teams to translate requirements into dependable data solutions.

Qualifications

• 5+ years of professional experience in data engineering, with ownership of production pipelines and large-scale data processing.
• Strong SQL expertise with complex joins, window functions, transformations, query tuning, and troubleshooting.
• Experience with Hive and the Hadoop ecosystem, including partitioning and distributed processing concepts.
• Proficiency in Python for data processing, automation, and maintainable production code.
• Hands-on experience with S3, Athena, EC2, and EMR on AWS.
• Strong experience with Airflow or other ETL/workflow tools, Airflow preferred.
• Experience implementing data quality checks, monitoring, error handling, and operations support.
• Ability to translate business requirements into technical designs and deliver solutions with minimal supervision.
• Strong documentation and communication skills for stakeholders.
• Experience with Jira and Confluence for task tracking and delivery coordination.
• Hands-on experience creating AI agents or AI-assisted automations.

Responsibilities

• Own data pipeline development: Design, develop, and maintain ETL/ELT pipelines for large-scale datasets across platforms.
• Lead technical design: Define processing logic, data models, integration approaches, and workflow dependencies.
• Develop and optimize data processing: Write SQL, Hive, and Python; use Spark where appropriate.
• Build and manage AWS workflows: Use S3, Athena, EC2, EMR; monitor and optimize resources.
• Manage workflow orchestration: Build automated workflows with Airflow, including scheduling, dependencies, retries, alerts.
• Ensure production reliability: Monitor pipelines, investigate failures, perform root cause analysis, implement improvements.
• Maintain data quality and governance: Validation checks, lineage, access controls across data lifecycle.
• Lead and mentor a small team: Guide 2-3 engineers, break requirements into tasks, review code and designs.
• Collaborate with stakeholders: Work with analytics, product, engineering, and data science to clarify requirements.
• Maintain technical documentation: Create/update solution designs, data flow diagrams, dictionaries, runbooks in Confluence.
• Apply AI to engineering workflows: Identify AI opportunities, build simple AI agents, validate outputs.
• Support agile delivery: Use Jira, contribute to sprint planning and technical reviews.

Skills

SQL expertise

Hive & Hadoop

Python

AWS

Airflow

Data quality

Documentation

Jira & Confluence

AI for engineering

Team mentoring

Tools

PySpark

Snowflake

OpenAI Codex

Snowflake Cortex

Job description

iScale Solutions in Makati, Philippines, is seeking an experienced data engineer to own end-to-end data pipelines, enabling reliable ingestion and processing of large-scale datasets across platforms.

You will design scalable architectures, implement robust QA, and leverage AWS services, Hive, Spark, and Airflow while collaborating with analysts, product, and data science teams to translate requirements into dependable data solutions.