Senior Hadoop & ETL Lead: Data Pipelines

Comunidade Metodista; West Chester (Chester County); On-site; USD 120,000 - 180,000; Full time

Job summary

Comunidade Metodista is seeking a Senior Data Engineer to design and optimize data platforms in a large relational and Hadoop-based environment. You will translate requirements into robust DB designs and build dashboards to support business needs.

You will translate raw data from multiple sources, optimize data pipelines, and collaborate with analysts to ensure accurate insights. Strong SQL, Hadoop, and scripting skills are essential for success.

Qualifications

• 10+ years of experience in IT and data environments.
• 2) 8+ years programming in large relational databases.
• 3) 6+ years SQL (Oracle preferred).
• 4) 3+ years SQL with Hadoop-based tech (Hive, Spark, Impala).
• 5) 3+ years of Linux shell / Python / Perl scripting.
• 6) Experience building data pipelines for large data sets.
• 7) Exposure to Kafka and Spark Streaming.
• 8) Deep understanding of relational DB concepts and performance.
• 9) 3+ years data analysis, modeling, docs creation.
• 10) Experience with Hadoop/Drill/Hive.
• 11) Experience with CVS/SVN/GIT.
• 12) Strong data warehousing and modeling practices.
• 13) Hadoop tuning (indexing/partitioning).
• 14) Knowledge of SDLC.
• 15) Experience with Tableau or similar dashboards.
• 16) Research and innovative problem-solving.
• 17) Advanced MS Word and Excel skills.

Responsibilities

• Interact with IT BAs or business partners to gather design information.
• 2) Design DBs and dashboards following requirements.
• 3) ETL raw data into aggregated datasets for projects.
• 4) Propose alternate data grouping and filtering for performance.
• 5) Provide frontline production support analyzing data and adjusting processes.
• 6) Work independently and in teams across multiple initiatives.
• 7) Resolve business process conflicts related to specs.
• 8) Report project statuses to IT management.

Skills

Excellent communication

Database tooling

SQL mastery

Hive

Spark

Impala

Python

Linux scripting

ETL development

Data modeling

Kafka

Spark Streaming

Tableau

Version control

Data warehousing

Performance tuning

SDLC knowledge

Education

10+ years experience

Tools

Hadoop

Hive

Drill

Git

SVN

CVS

Job description

Comunidade Metodista is seeking a Senior Data Engineer to design and optimize data platforms in a large relational and Hadoop-based environment. You will translate requirements into robust DB designs and build dashboards to support business needs.

You will translate raw data from multiple sources, optimize data pipelines, and collaborate with analysts to ensure accurate insights. Strong SQL, Hadoop, and scripting skills are essential for success.