Senior Data Engineer - Real-Time AWS Pipelines

ARC-One Solutions; United States; On-site; USD 100,000 - 157,000; Full time

Benefits offered by this job

Flexible work hours

Remote work with internet access

Travel as needed

Job summary

ARC-One Solutions is seeking a data engineering lead to manage and evolve an enterprise data lake and warehouse, ensuring secure and efficient data flow. The base salary range is provided, with pay determined by knowledge, skills, education, and location.

Responsibilities include designing scalable pipelines on AWS (S3, Glue, Redshift, Lambda, MWAA), developing batch and near real-time ETL/ELT workflows in Python/PySpark, and implementing governance and security controls.

Qualifications

• Bachelor's degree in computer-related field with 5+ years in data engineering.
• Experience building scalable AWS-based data platforms and pipelines using Lambda, Glue, Athena, S3, Redshift, DMS, MWAA, and Step Functions.
• Advanced proficiency in Python, SQL, and PySpark with hands-on ETL/ELT framework development.
• Experience implementing data quality, governance, and optimization practices in AWS.
• Strong communication to translate data concepts for stakeholders; healthcare/regulatory experience preferred.
• Experience with metadata, data lineage, observability, and data catalogs.

Responsibilities

• Design, implement, and maintain scalable data pipelines on AWS using S3, DMS, Glue, Lambda, MWAA, and Redshift.
• Develop robust batch and near real-time ETL/ELT workflows using Python and PySpark.
• Implement incremental/CDC mechanisms with restart, idempotency, and recovery capabilities.
• Introduce automated controls for completeness, accuracy, and lineage.
• Optimize Glue/Spark, Athena, Redshift, and S3 workloads via partitioning and cost-aware design.
• Build near real-time/event-driven pipelines with Kinesis/Kafka when needed.
• Enforce data security, least privilege, governance, and regulatory compliance.
• Monitor pipelines with CloudWatch; troubleshoot and resolve production incidents.
• Promote CI/CD, Git version control, testing, and cross-team collaboration.
• Document pipelines and operational procedures.

Skills

Python

SQL

PySpark

Data modeling

ETL/ELT design

Communication

Regulatory compliance

Education

Bachelor's degree in computer-related field

Tools

AWS Lambda

AWS Glue

Amazon Redshift

AWS S3

DMS

MWAA (Airflow)

Step Functions

Kinesis

Kafka

Git

CloudWatch

Job description

ARC-One Solutions is seeking a data engineering lead to manage and evolve an enterprise data lake and warehouse, ensuring secure and efficient data flow. The base salary range is provided, with pay determined by knowledge, skills, education, and location.

Responsibilities include designing scalable pipelines on AWS (S3, Glue, Redshift, Lambda, MWAA), developing batch and near real-time ETL/ELT workflows in Python/PySpark, and implementing governance and security controls.