Lead Data Engineer (Big Data Specialist)

Placements24; United States; Remote; USD 150,000 - 190,000; Full time

Benefits offered by this job

Stock options or equity

Comprehensive health insurance

Full remote with flexible schedule

Professional development opportunities

Job summary

Placements24 is seeking a visionary Lead Data Engineer to architect and scale data pipelines and infrastructure for advanced analytics and ML initiatives. You will lead a distributed team, drive data processing innovation, and ensure reliability and efficiency of the data ecosystem in a fully remote role.

You will mentor engineers, collaborate with data scientists and software engineers, and stay ahead of emerging Big Data technologies and best practices.

Qualifications

• Master's or Bachelor's degree in CS/Engineering or related field.
• 5+ years in data engineering with Big Data focus.
• Experience designing data pipelines with Spark.
• Deep knowledge of Hadoop, Hive, HDFS, Kafka.
• Programming: Python, Scala, or Java.
• Strong SQL and database knowledge (relational/NoSQL).
• Experience with cloud platforms (AWS/Azure/GCP) preferred.
• Excellent leadership and communication, remote collaboration.

Responsibilities

• Lead design and optimization of batch and real-time data pipelines.
• Build scalable, fault-tolerant data infrastructure using Spark, Hadoop, Kafka.
• Develop data warehousing and data lake solutions.
• Implement ETL/ELT processes from multiple sources.
• Ensure data quality, security, and governance across systems.
• Collaborate with data scientists, analysts, and engineers to meet data needs.
• Mentor junior data engineers and promote best practices.
• Automate data processes and monitor system performance.

Skills

Big Data

Data Pipelines

Distributed Systems

Python/Scala/Java

Leadership

Education

Master's or Bachelor's in CS/Engineering

Tools

Spark

Hadoop

Kafka

Hive

HDFS

Job description

About the Role

Our client, a cutting-edge data analytics firm, is seeking a visionary Lead Data Engineer with specialized expertise in Big Data technologies to join their distributed team. This pivotal role involves architecting, building, and maintaining scalable data pipelines and infrastructure that enable advanced analytics and machine learning initiatives. You will lead a team of talented data engineers, drive innovation in data processing, and ensure the reliability and efficiency of our client's data ecosystem. This is a fully remote position, offering the ultimate flexibility to contribute to critical projects from anywhere in the world. If you are passionate about harnessing the power of Big Data and have a proven track record of building robust data solutions, this is an exceptional opportunity to shape the future of data engineering.

Key Responsibilities

• Lead the design, development, and optimization of large-scale batch and real-time data pipelines.
• Build and manage robust, scalable, and fault-tolerant data infrastructure using Big Data technologies (e.g., Spark, Hadoop ecosystem, Kafka).
• Develop and maintain data warehousing solutions and data lakes.
• Implement ETL/ELT processes to ingest, transform, and load data from various sources.
• Ensure data quality, integrity, and security across all data systems.
• Collaborate with data scientists, analysts, and software engineers to understand data needs and deliver effective solutions.
• Mentor and guide junior data engineers, fostering best practices and technical excellence.
• Automate data processes and monitor system performance for optimal efficiency.
• Stay current with emerging Big Data technologies and best practices.

Requirements

• Master's or Bachelor's degree in Computer Science, Engineering, or a related quantitative field.
• 5+ years of experience in data engineering, with a strong focus on Big Data technologies.
• Proven experience designing and implementing data pipelines using frameworks like Apache Spark.
• In-depth knowledge of distributed systems and Big Data technologies (Hadoop, Hive, HDFS, Kafka).
• Proficiency in programming languages such as Python, Scala, or Java.
• Strong SQL skills and experience with various database technologies (relational and NoSQL).
• Experience with cloud data platforms (AWS, Azure, GCP) is highly desirable.
• Familiarity with data warehousing concepts and data modeling techniques.
• Excellent leadership, problem-solving, and communication skills.
• Ability to work effectively in a remote, collaborative team environment.

Benefits

• Highly competitive salary and performance-based bonuses.
• Comprehensive health, dental, and vision insurance.
• Generous stock options or equity grants.
• Full remote work flexibility, allowing you to work from anywhere.
• Opportunities for continuous learning and professional development.
• Access to the latest Big Data tools and technologies.
• Collaborative and innovative team culture.
• Regular virtual team events and professional networking opportunities.