Sr Data Engineer

Talentify; Charlotte (NC); On-site; USD 120,000 - 180,000; Full time

Job summary

Talentify seeks a Sr Data Engineer to join a data team focused on building scalable data integration and ETL pipelines. You will design and develop data/batch processing using Python, PySpark, and AWS services to deliver business value and support analytics across large data domains.

You will collaborate with business areas to define requirements, document designs, and implement robust solutions while adhering to SDLC and DevOps practices.

Qualifications

• Hands-on data engineering with Python and PySpark
• Experience with AWS services (Glue, Lambda, S3, RDS, MSK, Step Functions)
• Strong SQL and relational database skills (Postgres, SQL Server)
• Proficiency in design and development of ETL pipelines
• Experience with REST APIs and data integration patterns
• Familiarity with UNIX scripting and data modelling techniques

Responsibilities

• Provide scoping, estimating, planning, design, development and support for projects
• Develop and maintain ETL pipelines for large data domains
• Design data integration solutions using Python/PySpark and AWS services
• Document technical designs and present alternatives to clients
• Collaborate with analysts to prioritize requirements
• Follow SDLC processes and ensure quality and defect tracking
• Coordinate vendors and contractors as needed
• Stay updated on industry best practices and trends

Skills

Python

PySpark

AWS

ETL Development

SQL

Data Engineering

Unix scripts

REST APIs

CI/CD DevOps

GitHub

Jenkins

Agile/SCRUM

Data modeling

Tools

Control-M

Autosys

GitHub

Jenkins

Job description

Job Description:
The primary role of the Sr Data Engineer is to serve as a critical member of a data team, designing data integration solutions that deliver business value aligned with the company's objectives. They are responsible for the design and development of data/batch processing, data manipulation, data mining, and data extraction/transformation/loading into large data domains using Python/PySpark and AWS tools.

Responsibilities:

• Provide scoping, estimating, planning, design, development, and support services to a project.
• Identify and develop the technical detail design document.
• Work with developers and business areas to design, configure, deploy, and maintain custom ETL Infrastructure to support project initiatives.
• Design and develop data/batch processing, data manipulation, data mining, and data extraction/transformation/loading (ETL Pipelines) into large data domains.
• Document and present solution alternatives to clients, which support business processes and business objectives.
• Work with business analysts to understand and prioritize user requirements.
• Design, develop, test, and implement application code.
• Follow proper software development lifecycle processes and standards.
• Quality Analysis of the products, responsible for the Defect tracking and Classification.
• Track progress and intervene as needed to eliminate barriers and ensure delivery.
• Resolve or escalation problems and manage risk for both development and production support.
• Coordinate vendors and contractors for specific projects or systems.
• Maintain deep knowledge and awareness of technical & industry best practices and trends, especially in technology & methodologies.

Skills and Knowledge:

• Developer experience specifically focusing on Data Engineering.
• Hands-on experience in Development using Python and PySpark as an ETL tool.
• Experience in AWS services like Glue, Lambda, MSK (Kafka), S3, Step functions, RDS, EKS etc.
• Experience in Databases like Postgres, SQL Server, Oracle, Sybase, etc.
• Experience with SQL database programming, SQL performance tuning, relational model analysis, queries, stored procedures, views, functions, and triggers.
• Strong technical experience in Design (Mapping specifications, HLD, LLD), Development (Coding, Unit testing).
• Knowledge in developing UNIX scripts, Oracle SQL/PL-SQL.
• Experience with data models, data mining, data analysis, and data profiling.
• Experience in working with REST APIs.
• Experience with workload automation tools like Control-M, Autosys, etc.
• Good knowledge of CI/CD DevOps process and tools like GitHub, Jenkins.
• Strong experience with Agile/SCRUM methodology.
• Experience with other ETL tools (DataStage, Informatica, Pentaho, etc.).
• Knowledge in MDM, Data Warehouse, and Data Analytics.