Data Engineer - Python/PySpark & AWS ETL Specialist

Lancesoft; Charlotte (NC); On-site; USD 90,000 - 140,000; Full time

Job summary

Lancesoft in Charlotte, NC is seeking a Data Engineer to design data integration solutions and develop data/batch processing, data manipulation, data mining, and ETL into large data domains using Python/PySpark and AWS.

The role emphasizes REST APIs, SQL, CI/CD, and DevOps practices, with a focus on data warehousing and analytics. You''ll work on Agile/SCRUM teams and collaborate with data science and analytics stakeholders.

Qualifications

• Developer experience specifically focusing on Data Engineering.
• Hands-on experience in Development using Python and Pyspark as an ETL tool.
• Experience in AWS services like Glue, Lambda, MSK (Kafka), S3, Step functions, RDS, EKS etc.
• Experience in Databases like Postgres, SQL Server, Oracle, Sybase etc.
• Experience with SQL database programming, SQL performance tuning, relational model analysis, queries, stored procedures, views, functions, and triggers.
• Strong technical experience in Design (Mapping specifications, HLD, LLD), Development (Coding, Unit testing).
• Knowledge in developing UNIX scripts, Oracle SQL/PL-SQL.
• Experience with data models, data mining, data analysis and data profiling.
• Experience in working with REST API's.
• Experience in workload automation tools like Control-M, Autosys etc.
• Good knowledge in CI/CD DevOps process and tools like Bitbucket, GitHub, Jenkins.
• Strong experience with Agile/SCRUM methodology.
• Experience with other ETL tools (DataStage, Informatica, Pentaho, etc.).
• Knowledge in MDM, Data warehouse and Data Analytics.

Skills

Python

PySpark

AWS

SQL

Unix scripting

CI/CD

REST APIs

ETL tools

Data modeling

Data warehousing

Agile/SCRUM

Batch processing

Tools

Glue

Lambda

MSK

S3

Step Functions

RDS

EKS

Control-M

Autosys

Bitbucket

GitHub

Jenkins

DataStage

Informatica

Pentaho

Job description

Lancesoft in Charlotte, NC is seeking a Data Engineer to design data integration solutions and develop data/batch processing, data manipulation, data mining, and ETL into large data domains using Python/PySpark and AWS.

The role emphasizes REST APIs, SQL, CI/CD, and DevOps practices, with a focus on data warehousing and analytics. You''ll work on Agile/SCRUM teams and collaborate with data science and analytics stakeholders.