Sr. Hadoop/ETL Developer at Cloud Analytics Technologies LLC West Chester, PA

Comunidade Metodista; West Chester (Chester County); On-site; USD 120,000 - 180,000; Full time

Job summary

Comunidade Metodista is seeking a Senior Data Engineer to design and optimize data platforms in a large relational and Hadoop-based environment. You will translate requirements into robust DB designs and build dashboards to support business needs.

You will translate raw data from multiple sources, optimize data pipelines, and collaborate with analysts to ensure accurate insights. Strong SQL, Hadoop, and scripting skills are essential for success.

Qualifications

• 10+ years of experience in IT and data environments.
• 2) 8+ years programming in large relational databases.
• 3) 6+ years SQL (Oracle preferred).
• 4) 3+ years SQL with Hadoop-based tech (Hive, Spark, Impala).
• 5) 3+ years of Linux shell / Python / Perl scripting.
• 6) Experience building data pipelines for large data sets.
• 7) Exposure to Kafka and Spark Streaming.
• 8) Deep understanding of relational DB concepts and performance.
• 9) 3+ years data analysis, modeling, docs creation.
• 10) Experience with Hadoop/Drill/Hive.
• 11) Experience with CVS/SVN/GIT.
• 12) Strong data warehousing and modeling practices.
• 13) Hadoop tuning (indexing/partitioning).
• 14) Knowledge of SDLC.
• 15) Experience with Tableau or similar dashboards.
• 16) Research and innovative problem-solving.
• 17) Advanced MS Word and Excel skills.

Responsibilities

• Interact with IT BAs or business partners to gather design information.
• 2) Design DBs and dashboards following requirements.
• 3) ETL raw data into aggregated datasets for projects.
• 4) Propose alternate data grouping and filtering for performance.
• 5) Provide frontline production support analyzing data and adjusting processes.
• 6) Work independently and in teams across multiple initiatives.
• 7) Resolve business process conflicts related to specs.
• 8) Report project statuses to IT management.

Skills

Excellent communication

Database tooling

SQL mastery

Hive

Spark

Impala

Python

Linux scripting

ETL development

Data modeling

Kafka

Spark Streaming

Tableau

Version control

Data warehousing

Performance tuning

SDLC knowledge

Education

10+ years experience

Tools

Hadoop

Hive

Drill

Git

SVN

CVS

Job description

Job Description:

Technical/Functional Skills

• 1. EXCELLENT written, verbal and interpretation skills.
• 2. 8+ years of programming experience in large relational database environments
• 3. 6+ years of strong SQL (Oracle dialect preferred)
• 4. 3+ years of SQL using Hadoop-based technologies (Hive, Spark, Impala.)
• 5. 3+ years of scripting experience in Linux shell / python / Perl (understanding of Linux environment is a plus)
• 6. Experience developing data pipelines for large-scale data sets a plus.
• 7. Experience with streaming technologies such as Kafka and Spark Streaming a plus.
• 8. Excellent understanding of relational database concepts, impact of various design on performance
• 9. 3+ years of experience with data analysis, modeling, creating technical documentation
• 10. Experience using Hadoop / Drill / Hive
• 11. Experience with industry standard version control systems - like CVS or SVN or GIT
• 12. Strong experience with data warehousing concepts and data modeling best practices.
• 13. Strong knowledge of extremely large database optimization practices (indexing, partitioning, etc.) in Hadoop.
• 14. Familiar with all aspects of Software Development Life-Cycle.
• 15. Experience with other industry standard dashboard tools such as Tableau
• 16. Ability to research and come up with innovative solutions to problems with little to no up-front information.
• 17. Advanced skills in MS Word and Excel

Experience Required 10 yrs +

Roles & Responsibilities

• 1. Interacting with IT Business Analysts and / or our business partners to acquire the information necessary for product design.
• 2. Following written requirements to help design databases and develop dashboards to support our customers.
• 3. Translating raw data from multiple sources (ETL) into a single dataset at aggregate levels necessary to meet project needs.
• 4. Suggesting alternative ways to group, filter and sort aggregated data for optimal presentation and performance possibilities.
• 5. Providing front-line production support by analyzing customer questions, evaluating data sets to explain end results or quickly adjust processes based on consumer demands.
• 6. Working independently and with groups while engaging in multiple initiatives simultaneously in a fast paced IT environment.
• 7. Resolving business process conflicts as they relate to project specifications.
• 8. Reporting project statuses to IT Management staff as required