Remote | Big Data Engineer — $30–$80/hour

24-Mag Llc; New York (NY); Remote; USD 41,000 - 110,000; Part time

Job summary

24-MAG LLC is offering a part-time consulting opportunity for experienced Big Data Engineers to contribute technical expertise to an advanced AI development project involving large-scale data pipelines, distributed systems, databases, and modern data-engineering workflows.

Selected professionals will apply practical big-data engineering experience to complex data problems involving pipeline architecture, integration, transformation, distributed processing, storage, and system reliability.

Qualifications

• Experience in big-data engineering with large-scale pipelines and architectures.
• Strong Python skills for data processing, automation, and integration.
• Deep understanding of relational and NoSQL databases and performance tuning.
• Experience with distributed data-processing frameworks such as Hadoop, Spark, or Flink.

Responsibilities

• Design, build, and maintain scalable data pipelines and big-data architectures
• Develop workflows for ingesting, transforming, processing, and delivering large data volumes
• Translate requirements into robust data-engineering solutions
• Apply architectural patterns for distributed, high-volume data systems
• Ensure workflows are scalable, maintainable, and production-ready

Skills

Python

Big data

Distributed systems

ETL

SQL/NoSQL

Tools

Hadoop

Spark

Flink

Job description

We are sharing a specialised part-time consulting opportunity for experienced Big Data Engineers to contribute technical expertise to an advanced AI development project involving large-scale data pipelines, distributed systems, databases, and modern data-engineering workflows.

Selected professionals will apply practical big-data engineering experience to complex data problems involving pipeline architecture, integration, transformation, distributed processing, storage, and system reliability. The work requires strong Python skills, deep knowledge of database technologies, and the ability to design and evaluate scalable data solutions across modern engineering environments.

Key Responsibilities

Big Data Architecture & Pipeline Engineering

• Design, build, and maintain scalable data pipelines and big-data architectures
• Develop reliable workflows for ingesting, transforming, processing, and delivering large volumes of data
• Translate technical and business requirements into robust data-engineering solutions
• Apply appropriate architectural patterns for distributed and high-volume data systems
• Ensure data workflows remain scalable, maintainable, and suitable for production-oriented environments

Python, Data Integration & Processing

• Develop data-processing, automation, and integration solutions using Python
• Implement ETL and data-transformation workflows across structured and unstructured data sources
• Work with distributed processing technologies and modern big-data frameworks
• Build reproducible and efficient programmatic workflows for large-scale data operations
• Identify opportunities to improve processing efficiency, reliability, and maintainability

Databases, Storage & Performance

• Develop, manage, and optimise relational, NoSQL, distributed database, and storage systems
• Apply data-modelling and data-warehousing principles to support scalable analytical and operational workloads
• Monitor data-system performance and identify bottlenecks, failures, and reliability issues
• Troubleshoot performance, availability, and integration problems across distributed data environments
• Optimise storage, queries, and processing workflows for efficiency and resilience

Data Quality, Governance & Technical Collaboration

• Apply appropriate data-quality, security, and governance standards across engineering workflows
• Collaborate with technical and cross-functional stakeholders to understand data requirements and delivery priorities
• Document architectures, data flows, implementation decisions, and technical trade-offs
• Communicate complex engineering concepts clearly to both technical and non-technical audiences
• Support consistent engineering standards across remote and distributed project environments

Ideal Profile

• Proven professional experience in big-data engineering with hands-on responsibility for large-scale data pipelines and architectures
• Advanced proficiency in Python for data processing, automation, integration, and related engineering workflows
• Deep understanding of relational and NoSQL databases, including optimisation, management, and performance considerations
• Practical experience with distributed data-processing frameworks such as Hadoop, Spark, Flink, or comparable technologies
• Strong foundation in data modelling, ETL processes, data integration, and data-warehousing principles
• Ability to troubleshoot distributed systems, database performance, pipeline failures, and data-quality issues
• Excellent written and verbal communication skills
• Ability to explain technical concepts clearly to both engineering and non-technical stakeholders
• Strong attention to detail, initiative, and ability to work independently in remote environments
• Experience supporting fast-paced, startup-like, or globally distributed teams is advantageous
• Familiarity with cloud-based data platforms such as AWS, GCP, or Azure is advantageous
• Exposure to machine-learning operations, data-science workflows, or adjacent AI infrastructure is beneficial

Engagement Details

• Part-time independent contractor engagement
• Fully remote
• Compensation: $30-$80/hour
• Work will involve big-data architecture, pipeline development, Python-based processing, database optimisation, and distributed data systems
• Collaboration may involve technical, product, data, and other cross-functional stakeholders
• Project scope, workload, timing, and duration may evolve depending on project requirements
• Work must be completed without using confidential or proprietary information belonging to any employer, client, institution, or other third party

About the Platform

This opportunity is available through 24-MAG LLC. We connect experienced professionals with remote consulting opportunities across technical, evaluation, and project-based workstreams.

By submitting this application, you acknowledge that your information may be processed by 24-MAG LLC for recruitment and opportunity matching in accordance with our Privacy Policy: https://www.24-mag.com/privacy-policy