Data Engineer

Talentify; Charlotte (NC); On-site; USD 95,000 - 125,000; Full time

Job summary

Talentify is seeking a Data Engineer in Charlotte, NC for a 12-month contract. You will design and implement data integration pipelines using Python and PySpark, leveraging AWS services such as Glue, Lambda, and S3. Collaboration with developers and business teams will drive scalable data solutions.

The role requires experienc e in SQL, REST APIs, and CI/CD workflows with GitHub/Jenkins, following Agile/SCRUM practices to deliver high-quality data platforms and analytics capabilities.

Qualifications

• Proficient in Python and PySpark for ETL development.
• Strong understanding of AWS services (Glue, Lambda, S3, MSK, RDS, EKS).
• Experience with SQL databases and query tuning.
• Experience with REST APIs and data integration.
• Familiar with GitHub/Jenkins for CI/CD workflows.
• Knowledge of Agile/SCRUM practices.
• Exposure to external ETL tools (DataStage, Informatica, Pentaho).
• Understanding of data warehousing, MDM, and analytics.

Responsibilities

• Design and develop ETL pipelines for large data domains.
• Collaborate with developers and business areas to implement data solutions.
• Estimate, plan, and deliver data integration projects.
• Build and maintain data pipelines using Python and PySpark.
• Document technical designs and present solutions to clients.
• Manage risks and resolve issues in development and production.
• Coordinate vendors/contractors on projects.
• Adhere to SDLC and quality assurance practices.

Tools

Python (ETL)

PySpark

AWS

SQL

REST APIs

CI/CD (GitHub/Jenkins)

Unix scripting

Data Modeling

Job description

Position Details

Job Title

Data Engineer

Location

Charlotte, NC

Contract/Full-time

+12 Months contract with possible extension.

Job Description

The primary role of the Data Engineer is to function as a critical member of a data team by designing data integration solutions that deliver business value in line with the company’s objectives. They are responsible for the design and development of data/batch processing, data manipulation, data mining, and data extraction/transformation/loading into large data domains using Python/Pyspark and AWS tools.

Responsibilities

• Provide scoping, estimating, planning, design, development, and support services to a project.
• Identify and develop the technical detail design document.
• Work with developers and business areas to design, configure, deploy, and maintain custom ETL Infrastructure to support project initiatives.
• Design and develop data/batch processing, data manipulation, data mining, and data extraction/transformation/loading (ETL Pipelines) into large data domains.
• Document and present solution alternatives to clients, which support business processes and business objectives.
• Work with business analysts to understand and prioritize user requirements.
• Design, development, test, and implement application code.
• Follow proper software development lifecycle processes and standards.
• Quality Analysis of the products, responsible for the Defect tracking and Classification.
• Track progress and intervene as needed to eliminate barriers and ensure delivery.
• Resolve or elevate problems and manage risk for both development and production support.
• Coordinate vendors and contractors for specific projects or systems.
• Maintain deep knowledge and awareness of technical & industry best practices and trends, especially in technology & methodologies.

Skills and Knowledge

• Developer experience specifically focusing on Data Engineering.
• Hands-on experience in Development using Python and Pyspark as an ETL tool.
• Experience in AWS services like Glue, Lambda, MSK (Kafka), S3, Step functions, RDS, EKS etc.
• Experience in Databases like Postgres, SQL Server, Oracle, Sybase etc.
• Experience with SQL database programming, SQL performance tuning, relational model analysis, queries, stored procedures, views, functions, and triggers.
• Strong technical experience in Design (Mapping specifications, HLD, LLD), Development (Coding, Unit testing).
• Knowledge in developing UNIX scripts, Oracle SQL/PL-SQL.
• Experience with data models, data mining, data analysis and data profiling.
• Experience in working with REST API's.
• Experience with workload automation tools like Control-M, Autosys etc.
• Good knowledge in CI/CD DevOps process and tools like GitHub, Jenkins.
• Strong experience with Agile/SCRUM methodology.
• Experience with other ETL tools (DataStage, Informatica, Pentaho, etc.).
• Knowledge in MDM, Data warehouse and Data Analytics.