Senior Data Engineer

Compunnel, Inc.; Bentonville (AR); On-site; USD 60,000 - 80,000; Full time

Job summary

An established industry player is on the lookout for a Senior Data Engineer to join their dynamic team for a 12-month engagement. In this pivotal role, you'll design and develop cutting-edge big data applications, leveraging the latest open-source technologies while automating workflows and creating robust data pipelines. You'll have the opportunity to work with cloud platforms, particularly Google Cloud Platform, and mentor junior engineers, making a significant impact on your team's success. This role promises a collaborative environment where your expertise in data engineering will shine, and your contributions will help shape innovative solutions in the data landscape.

Qualifications

• 12+ years experience in data warehouse solutions and data products.
• 6+ years in distributed data processing platforms with Hadoop, Hive, Spark.

Responsibilities

• Design and develop big data applications using open-source technologies.
• Automate workflows and create data pipelines with Apache tools.

Skills

Data Engineering

Cloud Platforms

Workflow Orchestration

Programming (Python, Java, Scala)

Communication Skills

Education

Bachelor's degree in Computer Science

Tools

Apache Airflow

Apache Hive

Apache Spark

Google Cloud Platform (GCP)

JIRA

Job description

We are seeking a highly energetic and collaborative Senior Data Engineer for a 12-month engagement. The ideal candidate will design and develop big data applications using the latest open-source technologies, automate workflows, create data pipelines, and provide ongoing maintenance and enhancements. This role requires extensive experience in cloud platforms, data engineering, and workflow orchestration.

Key Responsibilities:

1. Design and develop big data applications using the latest open-source technologies
2. Work in an offshore model and deliver managed outcomes
3. Develop logical and physical data models for big data platforms
4. Automate workflows using Apache Airflow
5. Create data pipelines using Apache Hive, Apache Spark, and Apache Kafka
6. Provide ongoing maintenance and enhancements to existing systems and participate in rotational on-call support
7. Learn business domains and technology infrastructure quickly while sharing knowledge with the team
8. Mentor junior engineers and provide technical guidance
9. Lead daily standups and design reviews
10. Groom and prioritize backlog using JIRA
11. Act as the point of contact for assigned business domains

Required Qualifications:

1. 12+ years of hands-on experience in developing data warehouse solutions and data products
2. 6+ years of experience developing distributed data processing platforms with Hadoop, Hive, Spark, and Airflow
3. 5+ years of experience in modeling and designing schemas for data lakes or RDBMS platforms
4. 4+ years of recent hands-on experience with Google Cloud Platform (GCP)
5. Experience building data pipelines in GCP, including GCP Dataproc, GCS, and BigQuery
6. Proficiency in programming languages such as Python, Java, and Scala
7. Experience with scripting languages such as Perl and Shell
8. Expertise in processing and managing large-scale data sets (multi-TB/PB scale)
9. Exposure to test-driven development and automated testing frameworks
10. Background in Scrum/Agile development methodologies
11. Strong ability to manage multiple priorities with minimal supervision
12. Excellent verbal and written communication skills
13. Bachelor's degree in Computer Science or equivalent experience

Preferred Qualifications:

1. Experience with Gitflow and Atlassian products (BitBucket, JIRA, Confluence)
2. Familiarity with Continuous Integration tools such as Bamboo, Jenkins, or TFS

Location:

Bentonville, Arkansas, 72712