Data Consultant: ETL & Data Pipeline Architect

Particle41; New York (NY); On-site; USD 120,000 - 170,000; Full time

Job summary

Particle41 is seeking a versatile Data Consultant to design, build, and maintain data pipelines and infrastructure for client data needs in the United States. You will work end-to-end on data solutions, collaborating with cross-functional teams to deliver high-quality, scalable data delivery.

The role emphasizes designing ETL pipelines, data storage optimization, data integration, validation, and documentation while staying current with best practices and cloud platforms.

Qualifications

• Bachelor's degree in Computer Science, Engineering, or related field.
• Proven experience as a Data Engineer/Consultant with 3+ years.
• Proficiency in Python.
• Experience with SQL and NoSQL databases.
• Strong understanding of data warehousing, data lakes, and ETL.
• Familiarity with cloud data platforms (AWS, Azure, or GCP).

Responsibilities

• Design, develop, and maintain scalable ETL pipelines to process large data volumes from diverse sources.
• Build and optimize data storage solutions, such as data lakes and data warehouses, for efficient data retrieval.
• Integrate structured and unstructured data from various systems to create a unified view for analysis.
• Ensure data accuracy, consistency, and completeness through validation and cleansing processes.
• Maintain documentation for data processes, tools, and systems and promote best practices for workflows.

Skills

Python

SQL

NoSQL

Data warehousing

Flask

Pandas

Spark

Pyspark

Machine learning

OpenCV

scikit-learn

Git

Linux

Shell scripting

Education

Bachelor's degree in Computer Science, Engineering, or related field

Tools

ELK stack

Redis

Distributed task queues

Git

Shell scripting

Job description

Particle41 is seeking a versatile Data Consultant to design, build, and maintain data pipelines and infrastructure for client data needs in the United States. You will work end-to-end on data solutions, collaborating with cross-functional teams to deliver high-quality, scalable data delivery.

The role emphasizes designing ETL pipelines, data storage optimization, data integration, validation, and documentation while staying current with best practices and cloud platforms.