Hybrid Data Engineer -- Local candidates only

Creative Solutions Services, LLC; New York (NY); Hybrid; USD 140,000 - 190,000; Full time

Job summary

Creative Solutions Services, LLC is seeking a Senior Data Engineer to design scalable data platforms powering analytics and AI initiatives. You will build reliable, high-performance data ecosystems on AWS using Databricks, Delta Lake, Snowflake, Kafka, and Spark, enabling Generative AI capabilities.

Based in New York, NY, the role is hybrid with in-person interview and onboarding requirements. Collaborate with architects, analysts, and data scientists to deliver enterprise-grade data solutions

Qualifications

• Bachelor's or Master's in CS/Engineering/IS or related field.
• 8+ years in software/data engineering or big data.
• Enterprise-scale data pipelines design/implementation experience.
• Hands-on with Python, PySpark, Spark SQL, SQL, Kafka, Databricks, Delta Lake, Snowflake, Hive.
• Experience building on AWS cloud platforms.
• Understanding of CI/CD and Git workflows.
• Strong analytical and problem-solving abilities.

Responsibilities

• Design, build, and maintain scalable data pipelines for batch and real-time data.
• Implement Lakehouse architectures using Databricks, Delta Lake, Bronze/Silver/Gold.
• Develop processing with PySpark, Spark SQL, Kafka, AWS Kinesis.
• Architect AWS-based data solutions (S3, EMR, EC2, Redshift, Lambda, IAM, SQS).
• Optimize performance and ensure data quality, governance, and security.
• Develop orchestration with Airflow/Databricks Workflows; enable CI/CD automation.
• Assist migrations to cloud-native platforms and support Generative AI initiatives.

Skills

Analytical skills

Troubleshooting

Problem-solving

Stakeholder communication

Education

Bachelor's or Master's in CS/Engineering/IS

Tools

Python

PySpark

Spark SQL

SQL

Kafka

Databricks

Delta Lake

Snowflake

Hive

AWS

Git

CI/CD

Job description

This position requires a mandatory in person interview and in person onboarding.
Senior Data Engineer
Contract Length: Until 12/31/26 with possible extension
Location: New York, NY
Work Setup: Hybrid

We are seeking an experienced Senior Data Engineer to design, develop, and optimize scalable data platforms that power enterprise analytics, machine learning, and AI-driven solutions. This role will build reliable, high-performance data ecosystems using Databricks, Spark, Delta Lake, Snowflake, Kafka, and AWS while supporting modern Generative AI initiatives.

Senior Data Engineer Responsibilities

• Design, build, and maintain scalable batch and real-time data pipelines for structured, semi-structured, and unstructured data.
• Implement Lakehouse architectures using Databricks, Delta Lake, and Medallion (Bronze, Silver, Gold) design patterns.
• Develop distributed processing and streaming solutions using PySpark, Spark SQL, Kafka, Spark Structured Streaming, and AWS Kinesis.
• Architect AWS-based solutions using S3, EMR, EC2, Athena, Redshift, RDS, Lambda, IAM, SNS, and SQS.
• Optimize performance through partitioning, clustering, caching, query tuning, and cost-efficient storage and processing strategies.
• Ensure data quality, governance, lineage, security, and compliance across data ecosystems.
• Develop orchestration and scheduling frameworks using Airflow and Databricks Workflows and build CI/CD automation.
• Support migrations from traditional Hadoop and EMR environments to modern cloud-native platforms.
• Implement Generative AI solutions leveraging LLMs, RAG, Vector Databases, AI Agents, and Model Context Protocol (MCP).
• Collaborate with architects, analysts, data scientists, and business stakeholders to deliver enterprise-grade solutions.

Senior Data Engineer Required Qualifications

• Bachelor's or Master's degree in Computer Engineering, Computer Science, Information Systems, or a related field.
• 8+ years of experience in software engineering, data engineering, or big data platform development.
• Strong experience designing and implementing enterprise-scale data pipelines.
• Hands-on expertise with:
  • Python
  • PySpark
  • Spark SQL
  • SQL
  • Kafka
  • Databricks
  • Delta Lake
  • Snowflake
  • Hive
• Experience building data solutions on AWS cloud platforms.
• Strong understanding of distributed computing, data modeling, and large-scale data processing.
• Experience with Git-based development workflows and CI/CD practices.
• Excellent analytical, troubleshooting, and problem-solving skills.

Senior Data Engineer Preferred Qualifications

• Experience with real-time streaming architectures and event-driven systems.
• Knowledge of data governance, metadata management, and data quality frameworks.
• Experience with generative AI technologies including:
  • LLMs
  • RAG
  • Vector Databases
  • AI Agents
  • MCP integrations
• Experience developing developer productivity tools and AI-assisted engineering workflows.
• Exposure to enterprise supply chain, retail, healthcare, or manufacturing data domains.
• AWS certifications are highly preferred.

Senior Data Engineer Technical Skills

Python, Java, SQL, Shell Scripting, C/C++, PySpark, Spark SQL, Hive, Databricks, Delta Lake, Snowflake, Kafka, HBase, Sqoop, Apache Airflow, Databricks Workflows, Oozie, AWS, Tableau, Git, Docker, Splunk, IntelliJ IDEA, PyCharm, Cursor, Generative AI, LLMs, RAG, Agentic AI Systems, MCP, and Vector Databases.

Senior Data Engineer Preferred Certifications

AWS Certified Solutions Architect – Associate, AWS Certified Cloud Practitioner, and Databricks Certifications (preferred).

What Success Looks Like

The Senior Data Engineer will deliver highly scalable and reliable data pipelines, improve platform performance and cost efficiency, enable enterprise analytics and AI initiatives through trusted data products, modernize data platforms, and leverage AI technologies to enhance engineering productivity, automation, and innovation.

We look forward to reviewing your application!
#NoLinkedIn
#tech
IND123