Data Engineer

Highbrow LLC; Morris Plains (NJ); Hybrid; USD 90,000 - 120,000; Full time

Job summary

A leading tech firm is seeking a Data Engineer to work with business and technical teams in Morris Plains, NJ. The ideal candidate will have robust development experience in Spark, Py-Spark, Shell scripting, and Hadoop. Responsibilities include writing performant code for data transformations, implementing dev-ops pipelines, and participating in Agile processes. Knowledge of AWS and health care domains is a plus. This role starts remotely and offers a collaborative work environment.

Qualifications

• Strong development experience in Spark, Py-Spark, Shell scripting, Teradata, Hive and Hadoop.
• Experience with AWS services like S3, EC2, and Lambda is crucial.
• Knowledge of the health care domain will be considered a plus.

Responsibilities

• Understand requirements from business and technical leadership.
• Design and document according to specified requirements.
• Write performant code and queries for data extraction and transformations.
• Implement dev-ops pipelines to deploy code artifacts.
• Participate in agile planning and feedback sessions.

Skills

Spark

Py-Spark

Shell scripting

Teradata

Hive

Hadoop

Agile

AWS

Unix/Linux Shell scripting (KSH)

DevOps tools

Tools

Databricks

CA7 Enterprise Scheduler

Jira

Confluence

Job description

Job Title: Data Engineer

Job ID: 2023-11994

Job Location: Morris Plains, NJ (remote to start)

Job Travel Location(s):

# Positions: 2

Employment Type: W2

Candidate Constraints:

Duration: Long Term

# of Layers: 0

Work Eligibility: All Work Authorizations are Permitted

Key Technology: Spark, Py-Spark, Shell scripting, Teradata, Hive and Hadoop

Job Responsibilities

• Work with business and technical leadership to understand requirements.
• Design to the requirements and document the designs.
• Ability to write product-grade performant code for data extraction, transformations and loading using Spark, Py-Spark
• Do data modeling as needed for the requirements.
• Write performant queries using Teradata SQL, Hive SQL and Spark SQL against Teradata and Hive
• Implementing dev-ops pipelines to deploy code artifacts on to the designated platform/servers like AWS or Hadoop Edge Nodes
• Implement Hadoop job orchestration using Shell scripting, Apache Oozie, CA7 Enterprise Scheduler and Airflow
• Troubleshooting the issues, providing effective solutions and jobs monitoring in the production environment
• Participate in sprint planning sessions, refinement/story-grooming sessions, daily scrums, demos and retrospectives.

Skills and Experience Required

• Strong development experience in Spark, Py-Spark, Shell scripting, Teradata, Hive and Hadoop
• Experience of Ab Initio is a bonus.
• Strong experience in writing complex and effective SQLs (using Teradata SQL, Hive SQL and Spark SQL) and Stored Procedures
• Excellent work experience on Hadoop as data warehouse/Data Lake implementations
• Experience in Agile and working knowledge on DevOps tools (Git, Jenkins, Artifactory)
• Unix/Linux Shell scripting (KSH) and basic administration of Unix servers
• CA7 Enterprise Scheduler
• Experience with AWS (S3, EC2, SNS, SQS, Lambda, ECS, Glue, IAM, and CloudWatch)
• Databricks (Delta lake, Notebooks, Pipelines, cluster management, Azure/AWS integration)
• Experience in Jira and Confluence
• Exercises considerable creativity, foresight, and judgment in conceiving, planning, and delivering initiatives.
• Health care domain knowledge is a plus.