We are seeking a highly experienced Senior Cloudera ETL Developer with a minimum of 7 years of experience in designing, developing, and optimizing robust data pipelines. The ideal candidate will have a strong background in the Cloudera ecosystem, including extensive hands‑on experience with NiFi, Spark, and SQL for building and managing complex ETL processes. This role requires a proactive individual who can translate business requirements into efficient and scalable data solutions.
Key Responsibilities
- Design, develop, and maintain high-performance ETL pipelines using Cloudera technologies, primarily Apache NiFi and Apache Spark.
- Write efficient and complex SQL queries to extract, transform, and load data from various sources into data warehouses and data lakes.
- Develop and optimize Spark jobs for large-scale data processing, ensuring performance and cost-efficiency.
- Utilize Apache NiFi to build data flows, manage data ingestion, and orchestrate complex ETL processes.
- Collaborate with data engineers, data scientists, and business analysts to understand data requirements and deliver effective solutions.
- Troubleshoot and resolve issues related to data pipelines, performance bottlenecks, and data quality.
- Implement data governance, data quality, and data security best practices within ETL processes.
- Document ETL processes, data models, and technical specifications.
- Mentor junior developers and contribute to the overall technical growth of the team.
- Stay updated with the latest trends and technologies in big data and ETL development.
Required Skills and Qualifications
- 7+ years of professional experience in ETL development and data engineering.
- Proven expertise in Apache NiFi for designing and implementing data flows.
- Strong proficiency in Apache Spark (PySpark or Scala) for data processing and transformation.
- Advanced SQL skills with experience in complex query writing and optimization.
- Extensive experience with the Cloudera ecosystem (e.g., HDFS, Hive, Impala).
- Solid understanding of data warehousing concepts, data modeling, and database design.
- Experience with scripting languages such as Python or Bash.
- Excellent problem-solving and analytical skills.
- Strong communication and collaboration abilities.
Preferred Qualifications
- Experience with other big data technologies (e.g., Kafka, HDFS).
- Familiarity with cloud-based data platforms (e.g., AWS, Azure, GCP).
- Experience with CI/CD pipelines and version control systems (e.g., Git).
- Knowledge of data visualization tools.
Skills
GIT Python AWS GCP Azure analytical skills SQL PySpark CI/CD pipelines communication Scala Problem Solving Bash Collaboration Data Visualization Data Warehousing Data Modeling apache spark hdfs database design kafka impala apache nifi Hive Data Engineering ETL Development Cloudera ecosystem