Data Engineer

Acestack; Charlotte (NC); On-site; USD 120,000 - 180,000; Full time

Job summary

Acestack in Charlotte, NC is seeking an experienced Data Engineer to design, build and optimize large-scale data pipelines. You will work under supervision from a senior domain expert, collaborating with Business Partners, Product Owner, Scrum Master, Development Team and Data Scientists to deliver robust data solutions.

The ideal candidate has 8+ years of hands-on data engineering or software development, strong experience with Hadoop/Cloudera, Hive, Spark, Python, SQL and UNIX, plus cloud

Qualifications

• 8+ years hands-on data engineer or software development experience.
• Onsite experience in Charlotte, NC (minimum 5 years).
• Strong background with Hadoop, Cloudera, Hive, Spark and Pispark.
• Proficient in Python, SQL and UNIX.
• Experience with cloud services such as AWS or Azure.
• Exposure to data stores like HIVE, HBase, Elastic; Bitbucket and CI/CD pipelines.
• Ability to lead junior engineers toward project delivery.
• Experience in cyber security domain, preferably vulnerability data management.

Responsibilities

• Ensure data solution designs are consistent across flows and systems.
• Conduct prototyping and evaluation of new toolsets.
• Use multiple architectural components for client requirements.
• Maintain and transform data for operational and analytics systems.
• Collaborate with dev teams to validate data architecture feasibility.
• Assemble large, complex data sets meeting requirements.
• Build data pipelines enabling faster, data-informed decisions.
• Contribute to test suites and triage issues found in tests.
• Document deployment, maintenance and business functionality details.
• Adhere to CI/CD release processes and timelines.
• Identify gaps in data management and plan improvements.
• Lead concept testing and synthesize results to improve analytics.
• Define KPIs and internal controls; mentor junior engineers.

Skills

8+ years data engineering

Hadoop

Cloudera

Python

SQL

UNIX

Hive

Spark

Bitbucket

CI/CD

AWS/Azure

Data pipelines

Leadership

Cyber security domain

Tools

Hadoop ecosystem tools

Cloud platforms

Data stores (HBase, Elastic)

Job description

Role: Data Engineer

Location: Charlotte, NC Onsite

Fulltime Permanent

Job Description

Skill: IT Security Engineer

Must Have Technical/Functional Skills:

Objective of Role:

The candidate is expected to have knowledge in the cyber security domain.

Primary Level of Engagement:

Works as a team member under supervision from a more senior domain expert.

Primary Interactions:

• Business partners and stakeholders
• Product Owner
• Scrum Master
• Development Team
• Feature Lead
• Architect
• Domain Community of Practice
• Data Scientists

Required Skills:

• - Minimum 8+ years hands-on data engineer/Software development exp (at least 5 years of onsite exp)
• - Experience with big data processing systems like Hadoop, Cloudera etc
• - Experience with one or more of the programming languages like Python (Modular python programming exp)
• - Strong SQL and UNIX experience
• - Hadoop Hive, Spark , Pispark
• Proven experience with technology leadership guiding junior engineers towards project completion and deliveries
• Experience with cyber security domain, preferably vulnerability data management
• 8+ years of hands-on data engineer/software development
• Ability to switch from technology leadership to hands on coding when required
• Experience with big data processing systems like Hadoop, Cloudera, etc.
• Experience with Cloud Services\Technologies like AWS, Azure, etc.
• Experience with one or more of the programming languages like Python
• Strong SQL and UNIX experience
• Experience with data stores like HIVE, HBase Elastic
• Experience with code versioning systems like Bitbucket
• Experience with CICD pipelines
• Good knowledge on Data engineering principles and SDLC process
• Ability to detect early and articulate with business partners the challenges and gaps in requirements

Experience: Minimum 8+ years

Roles & Responsibilities:

• Ensure that the design and engineering approach for complex data solutions is consistent across multiple flows and systems.
• Conduct research, designs prototyping and other exploration activities such as evaluating new toolsets.
• Utilize multiple architectural components in design and development of client requirements.
• Maintain, improve, clean and manipulate data for the operational and/or analytics data systems.
• Constantly looking for better ways of solving technical problems and designing the solution without being afraid of challenging the status quo.
• Collaborate with development teams to understand data requirements and ensure the data architecture is feasible to implement and subsequently implemented accurately.
• Assemble large, complex data sets that meet functional/ non-functional requirements.
• Build processes supporting data transformation, data structures, metadata, data quality controls, dependency and workload management.
• Define and build data pipelines that enable faster, better, data-informed decision-making within the business.
• Contribute to existing test suites (integration, regression, performance), analyze test reports, identify any test issues/errors, and triage the underlying cause.
• Document and communicate required information for deployment, maintenance, support, and business functionality.
• Adhere to team delivery/release process and cadence pertaining to code deployment and release
• Identify gaps in data management standards adherence and work with appropriate partners to develop plans to close gaps.
• Lead concept and experimentation testing, and synthesize the results to validate and improve analytical solution.
• Define key performance indicators and internal controls.
• Mentor more junior Data Engineers and coach team members in the delivery/release CI-CD events (for example, branching timelines, pull requests, issue triage, merge/conflict resolution, and release notes).