Senior AWS Data Engineer: Lakehouse & AI-Driven Pipelines

Global Business Ser. 4u; New York (NY); On-site; USD 140,000 - 190,000; Full time

Job summary

Global Business Ser. 4u is seeking a Senior Data Engineer with 8+ years of data and software engineering experience. The role focuses on designing and optimizing enterprise-scale batch and real-time pipelines for analytics and AI-driven solutions.

You will implement Lakehouse architectures using Delta Lake and Databricks, build high-volume streaming with Kafka and Spark Streaming, and deploy cloud-native platforms on AWS. Strong CI/CD, governance, and cost optimization are essential.

Qualifications

• 8+ years of experience in data engineering, software engineering, or big data platform development.
• Bachelor’s or Master’s degree in Computer Science, Engineering, Information Systems, or related field.
• Design, develop, and optimize enterprise-scale batch and real-time data pipelines.
• Hands-on with Spark, PySpark, Spark SQL, Python, SQL, Databricks, Delta Lake, Snowflake.
• Experience building Lakehouse architectures using Delta Lake and Medallion pattern.
• Develop streaming data solutions using Kafka, Spark Structured Streaming, AWS Kinesis.
• Architect cloud-native data platforms on AWS (S3, EMR, Redshift, Lambda, IAM, etc.).
• CI/CD and data orchestration with Airflow, Databricks Workflows, Git, Docker, Oozie.
• Ensure data quality, governance, security, lineage, and cost optimization.
• Familiarity with Generative AI, LLMs, Vector DBs, MCP is a plus.

Responsibilities

• Design, build, and optimize enterprise-grade batch and real-time data pipelines.
• Develop lakehouse architectures with Delta Lake, Databricks, and Medallion pattern.
• Implement streaming data solutions using Kafka and Spark Streaming.
• Collaborate on AI/ML analytics and AI-driven data platforms.
• Enforce data quality, governance, and cost controls across platforms.

Skills

Apache Spark

PySpark

Spark SQL

Python

SQL

Databricks

Delta Lake

Snowflake

Kafka

Spark Streaming

AWS Kinesis

Airflow

Databricks Workflows

Git

Docker

Oozie

Java

Shell Scripting

Hive

HBase

Sqoop

Tableau

Splunk

LLMs

Education

Bachelor’s/Master’s degree in Computer Science, Engineering, Information Systems, or related field

Tools

AWS

Job description

Global Business Ser. 4u is seeking a Senior Data Engineer with 8+ years of data and software engineering experience. The role focuses on designing and optimizing enterprise-scale batch and real-time pipelines for analytics and AI-driven solutions.

You will implement Lakehouse architectures using Delta Lake and Databricks, build high-volume streaming with Kafka and Spark Streaming, and deploy cloud-native platforms on AWS. Strong CI/CD, governance, and cost optimization are essential.