Big Data Engineer

Cynet Systems Inc; Vancouver; On-site; CAD 90,000 - 130,000; Full time

Job summary

A leading company in the tech sector is seeking a Big Data Architect to design and maintain scalable architectures and data systems. The role involves implementing data processing pipelines, optimizing performance, and collaborating with various stakeholders. Ideal candidates will have a strong background in big data technologies and experience in managing projects and teams.

Qualifications

• Experience with Hadoop, Spark, and Kafka for data processing.
• Familiarity with Docker and Kubernetes for application deployment.
• Knowledge of data security practices and compliance.

Responsibilities

• Design, develop, and maintain scalable big data architectures.
• Implement data processing pipelines using big data technologies.
• Collaborate with data scientists and analysts to deliver solutions.

Skills

Data Structures and Algorithms

Distributed Systems

Data Integration

Performance Optimization

Containerization

Version Control

Security

Machine Learning

Job description

Job Description:

Responsibilities:

• Design, develop, and maintain scalable big data architectures and systems.
• Implement data processing pipelines using technologies such as Hadoop, Spark, and Kafka.
• Optimize data storage and retrieval processes to ensure high performance and reliability.
• Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and deliver solutions.
• Perform data modeling, mining, and production processes to support business needs.
• Ensure data quality, governance, and security across all data systems.
• Stay updated with the latest trends and advancements in big data technologies.
• Experience with real-time data processing and stream analytics.
• Knowledge of advanced analytics and data visualization tools.
• Knowledge of DevOps practices and tools for continuous integration and deployment.
• Experience in managing big data projects and leading technical teams.

Technical Skills:

• Data Structures and Algorithms: Expertise in designing efficient data structures and algorithms for big data processing.
• Distributed Systems: Experience with distributed computing frameworks and parallel processing.
• Data Integration: Proficiency in integrating data from various sources and ensuring data consistency.
• Performance Optimization: Skills in optimizing data processing and storage for high performance.
• Containerization: Experience with Docker and Kubernetes for deploying big data applications.
• Version Control: Proficiency in using Git for collaborative development.
• Security: Knowledge of data security practices and compliance with regulatory standards.
• Machine Learning: Familiarity with machine learning techniques and their application in big data environments.