AWS Data Engineer

Global Business Ser. 4u; New York (NY); On-site; USD 140,000 - 190,000; Full time

Job summary

Global Business Ser. 4u is seeking a Senior Data Engineer with 8+ years of data and software engineering experience. The role focuses on designing and optimizing enterprise-scale batch and real-time pipelines for analytics and AI-driven solutions.

You will implement Lakehouse architectures using Delta Lake and Databricks, build high-volume streaming with Kafka and Spark Streaming, and deploy cloud-native platforms on AWS. Strong CI/CD, governance, and cost optimization are essential.

Qualifications

• 8+ years of experience in data engineering, software engineering, or big data platform development.
• Bachelor’s or Master’s degree in Computer Science, Engineering, Information Systems, or related field.
• Design, develop, and optimize enterprise-scale batch and real-time data pipelines.
• Hands-on with Spark, PySpark, Spark SQL, Python, SQL, Databricks, Delta Lake, Snowflake.
• Experience building Lakehouse architectures using Delta Lake and Medallion pattern.
• Develop streaming data solutions using Kafka, Spark Structured Streaming, AWS Kinesis.
• Architect cloud-native data platforms on AWS (S3, EMR, Redshift, Lambda, IAM, etc.).
• CI/CD and data orchestration with Airflow, Databricks Workflows, Git, Docker, Oozie.
• Ensure data quality, governance, security, lineage, and cost optimization.
• Familiarity with Generative AI, LLMs, Vector DBs, MCP is a plus.

Responsibilities

• Design, build, and optimize enterprise-grade batch and real-time data pipelines.
• Develop lakehouse architectures with Delta Lake, Databricks, and Medallion pattern.
• Implement streaming data solutions using Kafka and Spark Streaming.
• Collaborate on AI/ML analytics and AI-driven data platforms.
• Enforce data quality, governance, and cost controls across platforms.

Skills

Apache Spark

PySpark

Spark SQL

Python

SQL

Databricks

Delta Lake

Snowflake

Kafka

Spark Streaming

AWS Kinesis

Airflow

Databricks Workflows

Git

Docker

Oozie

Java

Shell Scripting

Hive

HBase

Sqoop

Tableau

Splunk

LLMs

Education

Bachelor’s/Master’s degree in Computer Science, Engineering, Information Systems, or related field

Tools

AWS

Job description

• Seeking a Senior Data Engineer with 8+ years of experience in data engineering, software engineering, or big data platform development, with a Bachelor’s/Master’s degree in Computer Science, Engineering, Information Systems, or a related field.
• Design, develop, and optimize enterprise-scale batch and real-time data pipelines supporting analytics, machine learning, reporting, and AI-driven solutions.
• Demonstrate strong hands-on expertise in Apache Spark, PySpark, Spark SQL, Python, SQL, Databricks, Delta Lake, and Snowflake.
• Build modern Lakehouse architectures using Databricks, Delta Lake, and Medallion Architecture (Bronze, Silver, Gold) for scalable data processing.
• Develop high-volume, low-latency streaming and distributed data solutions using Kafka, Spark Structured Streaming, AWS Kinesis, and related technologies.
• Architect and implement cloud-native data platforms on AWS, leveraging services such as S3, EMR, EC2, Athena, Redshift, RDS, Lambda, IAM, SNS, and SQS.
• Optimize large-scale data workloads through partitioning, clustering, caching, query tuning, distributed computing, data modeling, and performance optimization.
• Build data orchestration, automation, and CI/CD frameworks using Apache Airflow, Databricks Workflows, Git, Docker, and Oozie while supporting monitoring and operational reliability.
• Ensure enterprise data platforms meet requirements for data quality, governance, metadata, lineage, security, compliance, fault tolerance, and cost optimization.
• Apply Generative AI, LLMs, RAG, Vector Databases, AI Agents, Agentic AI, and Model Context Protocol (MCP) to build intelligent data platforms and AI-assisted engineering solutions.
• Experience with Azure AI, Microsoft Copilot Studio, Gemini Enterprise, Spark AI development platforms/frameworks/tools, and AI-assisted developer productivity solutions is highly preferred.
• Experience with Java, Shell Scripting, Hive, HBase, Sqoop, Tableau, Splunk, and AWS/Databricks certifications is a plus; domain exposure in supply chain, retail, healthcare, or manufacturing is preferred.