Data Engineer

Factspan; Atlanta (GA); On-site; USD 120,000 - 170,000; Full time

Job summary

Factspan seeks a senior Data Engineer for an onsite Atlanta role to support a Java Spring Boot based data platform. You will design scalable pipelines and collaborate across teams to ensure production readiness.

You will build data ingestion, transformation, and orchestration on Google Cloud (BigQuery, Dataflow, Cloud Composer) and Airflow, while implementing data quality checks and performance tuning.

Qualifications

• Must have expertise in Java-based Spring Boot applications.
• Strong hands-on experience with GCP services: BigQuery, Dataflow, GCS, Cloud Composer.
• Proficient in SQL for data profiling and optimization.
• Proficient in Python for pipeline automation and orchestration.
• Experience building batch and streaming data pipelines.

Responsibilities

• Design and develop data ingestion, transformation, and orchestration pipelines using GCP and Airflow.
• Implement and maintain data integrations between MongoDB and BigQuery for performance and scalability.
• Build and optimize ETL/ELT workflows for warehousing.
• Implement data quality checks, monitoring, and alerting frameworks.
• Translate requirements into data pipelines with Lead DE and SME.
• Support performance tuning, data reconciliation, and defect resolution during testing.
• Collaborate with QA for end-to-end validation and production readiness.
• Participate in daily Agile ceremonies and sprint planning.

Skills

Java

Spring Boot

Google Cloud Platform

BigQuery

SQL

Python

Airflow

Apache Beam

Education

Bachelor's or Master's in CS/IT

Job description

Location: Atlanta (Onsite)

Experience: 8-10 years

Employment Type: Full Time

About the Project

This role is part of the MarTech Data Engineering POD supporting a Client Facing Java based Spring Boot application. The initiative aims to build a scalable data warehouse solution which aggregates, transforms and loads sales data into MongoDB from Big Query with the help of Airflow and Dataflow. Finally, publish data with the help of Spring Boot application.

Must have:

Tech: Java, Spring Boot, Google Cloud Platform, Big Query, SQL, Python, Airflow, Apache Beam

Key Responsibilities:

• Design and develop data ingestion, transformation, and orchestration pipelines using Google Cloud Platform (BigQuery, Dataflow, Cloud Composer) and Airflow.
• Implement and maintain data integrations between MongoDB and Big Query, ensuring high performance and scalability.
• Build and optimize ETL/ELT workflows to support data orchestration, transformation and warehousing.
• Implement data quality checks, monitoring, and exception handling frameworks to ensure clean and reliable data.
• Work closely with Lead DE and SME to translate Requirements into data pipelines and orchestration rules.
• Support performance tuning, data reconciliation, and defect resolution during testing and hypercare phases.
• Collaborate with QA team for end-to-end test validation and ensure production readiness.
• Participate in daily Agile ceremonies, sprint reviews, and deployment planning.

Required Technical Skills:

• Must have expertise in Java based Spring Boot Application.
• Strong hands-on experience in Google Cloud Platform services: BigQuery, Dataflow, GCS, Cloud Composer (Airflow).
• Proficiency in SQL for data profiling, transformation, and performance optimization.
• Proficiency in Python for pipeline automation, scripting, and orchestration.
• Experience building batch and streaming data pipelines.
• Exposure to working with marketing or customer engagement platforms is a plus.
• Understanding data quality, logging, monitoring, and alerting best practices.

.

Education:

Bachelor’s or Master’s degree in Computer Science, Information Technology, or a related field.