Remote Data Automation Engineer — AWS & PySpark

Equiliem; Washington (District of Columbia); Remote; USD 76,000 - 83,000; Full time

Benefits offered by this job

Medical Insurance

Vision & Dental Insurance

Life Insurance

Matching 401(k)

Commuter Benefits

Employee Discounts & Rewards

Payroll Payment Options

Job summary

Equiliem is seeking a Data Automation Engineer to design and manage scalable cloud-based data pipelines using Python, SQL, AWS, and Spark. The role focuses on production data engineering, cloud integration, and CI/CD in an enterprise setting with Generative AI enhancements.

The position is remote with quarterly travel to Gaithersburg, MD, offering a 40-hour work week and a competitive hourly rate. A Public Trust clearance may be required.

Qualifications

• 5+ years of data engineering or related software development.
• Hands-on production data pipelines design and troubleshooting.
• Strong Python and SQL development skills.
• Experience with AWS data services (S3, Glue, Lambda, EMR).
• Spark/PySpark experience for data transformation.
• Familiarity with CI/CD, Git, and Agile methods.
• Ability to obtain a Public Trust clearance.

Responsibilities

• Design, develop, and maintain scalable ETL/ELT data pipelines.
• Build automated ingestion, transformation, and integration workflows.
• Develop solutions using AWS S3, Glue, Lambda, EMR, and Step Functions.
• Create batch and near-real-time pipelines using Kafka or Flume.
• Integrate Generative AI and LLM services into Python workflows.
• Develop integrations between AWS, Azure, SQL, APIs, and external sources.
• Troubleshoot data-processing issues and optimize performance.
• Implement CI/CD with GitHub, Jenkins, or Azure DevOps.
• Maintain security, integrity, and compliance of data.
• Collaborate with engineering teams and participate in Agile work.
• Document solutions, data workflows, and procedures.

Skills

Python

SQL

AWS

Spark PySpark

Education

Bachelor's degree in Computer Science or Engineering

Tools

S3

Glue

Lambda

EMR

Step Functions

Kafka

Flume

Docker

Kubernetes

Job description

Equiliem is seeking a Data Automation Engineer to design and manage scalable cloud-based data pipelines using Python, SQL, AWS, and Spark. The role focuses on production data engineering, cloud integration, and CI/CD in an enterprise setting with Generative AI enhancements.

The position is remote with quarterly travel to Gaithersburg, MD, offering a 40-hour work week and a competitive hourly rate. A Public Trust clearance may be required.