Remote | Data Engineer — $140,000–$180,000/year

24-Mag Llc; Northern, New York (KY, NY); Hybrid; USD 140,000 - 180,000; Full time

Job summary

24-MAG LLC is offering a full-time Data Engineer role for someone with Python, SQL, Spark, and AWS expertise to build and operate AI-ready data infrastructure.

The position focuses on distributed data pipelines, cloud architectures, data quality, and collaboration with AI researchers; fully remote with compensation $140,000–$180,000 per year.

Qualifications

• Experience in data engineering and distributed data systems.
• Proficient in Python and SQL.
• Hands-on with Spark and AWS data services.
• Familiar with orchestration, monitoring, data quality.

Responsibilities

• Design, build, and maintain large-scale data pipelines.
• Develop distributed processing workflows with Spark.
• Design scalable AWS-based data architectures.
• Write efficient Python and SQL for production processing.
• Support analytics, experimentation, and AI/ML workflows.

Skills

Python

SQL

Apache Spark

AWS

Distributed data systems

Data architecture

Data quality

AI/ML exposure

LLM datasets

Data visualization

Tools

Matplotlib

Seaborn

Plotly

Job description

We are sharing a full-time opportunity for an experienced Data Engineer with strong expertise in Python, SQL, Apache Spark, AWS, distributed data processing, and scalable data architecture to build and operate infrastructure supporting AI-driven products and research initiatives.

The role will focus on designing and scaling distributed data pipelines, managing large datasets across cloud environments, and building reliable systems for analytics, experimentation, and model development.

Key Responsibilities

Data Pipelines & Distributed Processing

• Design, build, and maintain large-scale pipelines for structured and unstructured data
• Develop distributed processing workflows using Apache Spark or comparable frameworks
• Optimise transformations, partitioning strategies, and computational workloads
• Identify and resolve performance bottlenecks across high-volume data systems
• Support downstream analytics, experimentation, and model-development requirements

Cloud Architecture & Data Engineering

• Design scalable AWS-based data architectures across SQL and NoSQL systems
• Build reliable ingestion, transformation, storage, and distribution workflows
• Write efficient Python and SQL for production data processing
• Evaluate storage and database technologies against workload requirements
• Improve scalability, maintainability, accessibility, and operational efficiency

Data Quality, Reliability & AI Support

• Implement monitoring, validation, and automation across data workflows
• Identify failures, anomalies, and data-quality issues
• Maintain integrity and reliability throughout pipelines and storage layers
• Collaborate with AI researchers, data scientists, and engineering teams
• Support data infrastructure for AI/ML training, evaluation, and experimentation

Ideal Profile

• Strong professional experience in data engineering or distributed data systems
• Advanced proficiency in Python and SQL
• Hands-on experience with Apache Spark or comparable distributed-processing frameworks
• Strong experience with AWS data services and cloud-native architecture
• Experience with SQL and NoSQL databases
• Demonstrated experience processing large-scale datasets
• Strong understanding of partitioning, performance optimisation, and scalable architecture
• Familiarity with orchestration, automation, monitoring, and data-quality workflows
• Exposure to AI/ML or research environments is advantageous
• Familiarity with LLM training, evaluation, or experimentation datasets is beneficial
• Experience with data-visualisation tools such as Matplotlib, Seaborn, or Plotly is a plus

Engagement Details

• Full-time engagement
• Fully remote
• Base compensation: $140,000–$180,000/year
• Work will involve Python, SQL, Apache Spark, AWS, distributed data processing, and scalable data architecture
• Responsibilities will span data ingestion, transformation, storage, monitoring, and operational reliability
• The role may support AI/ML experimentation, model-development workflows, and LLM-related data infrastructure
• Data volumes, infrastructure requirements, and technical priorities may evolve as products and research initiatives scale

About the Platform

This opportunity is available through 24-MAG LLC. We connect experienced professionals with remote consulting opportunities across technical, evaluation, and project-based workstreams.

By submitting this application, you acknowledge that your information may be processed by 24-MAG LLC for recruitment and opportunity matching in accordance with our Privacy Policy: https://www.24-mag.com/privacy-policy