Data Engineer

Siri InfoSolutions Inc; Atlanta (GA); On-site; USD 120,000 - 150,000; Full time

Job summary

Siri InfoSolutions Inc is seeking a Data Engineer for our Atlanta, GA office. The role focuses on building scalable data pipelines, performing ETL, and enabling analytics across mountain data sources.

You will work with Spark, Kafka, Hadoop, and cloud services to deliver robust data solutions. You will collaborate with data scientists and BI teams, ensure data quality, and contribute to governance and best practices.

Qualifications

• Experience designing scalable data pipelines and ETL solutions.
• Strong SQL querying and data modeling capabilities.
• In-depth knowledge of data ingestion, transformation, cleansing and validation.

Responsibilities

• Design, develop, and maintain scalable data pipelines and ETL solutions.
• Build and optimize data ingestion frameworks for structured and unstructured data.
• Implement data quality, validation, standardization, and governance processes.
• Develop data infrastructure integrating multiple heterogeneous data sources.
• Support analytics, reporting, ML, and BI initiatives.
• Collaborate with Data Scientists and BI teams to prepare data for analytics.
• Lead and mentor junior data engineers.

Skills

Databricks

Python

PySpark

Spark SQL

Apache Spark

ETL/Data Pipeline Development

Hadoop

Kafka

Hive

HDFS

Sqoop

SQL querying

Data ingestion

Cloud platforms (AWS/GCP)

Data warehouse

Kafka Streaming

Agile methodologies

Tools

Tableau

Power BI

Spotfire

OAC

Git

IBM DataStage

Tibco

Job description

Role: Data Engineer
Location: Atlanta, GA
Term: Full Time

Job Description

Must Have Technical/Functional Skills

Must Have Skills:

• Databricks (Notebooks, Delta Lake, Delta Live Tables, Unity Catalog)
• Python and PySpark
• Spark SQL and Apache Spark
• ETL/Data Pipeline Development
• Big Data technologies: Hadoop, Spark, Kafka, Hive, HDFS, Sqoop
• Strong SQL querying and data modeling
• Data ingestion, transformation, cleansing, and validation
• Cloud platforms (AWS/GCP)
• Data warehouse and large-scale data processing experience
• Experience with streaming data using Kafka and Spark Streaming
• Agile methodologies (Scrum, Kanban, SAFe)

Preferred Skills:

• Scala / Scala-Spark
• IBM DataStage, Tibco, or similar ETL tools
• AWS services: S3, Glue, Lambda, Kinesis/MSK
• BI Tools: Tableau, Power BI, Spotfire, OAC
• Databases: Teradata, DB2, BigQuery, Mainframe
• Git version control
• Data governance and retention modeling

Roles & Responsibilities

• Design, develop, and maintain scalable data pipelines and ETL solutions.
• Build and optimize data ingestion frameworks for structured and unstructured data.
• Implement data quality, validation, standardization, and governance processes.
• Develop data infrastructure integrating multiple heterogeneous data sources.
• Support analytics, reporting, machine learning, and business intelligence initiatives.
• Work with Data Scientists and BI teams to prepare and curate data for analytics.
• Develop scalable solutions for Big Data processing and streaming workloads.
• Analyze data and provide actionable insights and recommendations to business stakeholders.
• Gather requirements from business teams and translate them into technical solutions.
• Lead and mentor junior data engineers.
• Collaborate across multiple projects as the Data Engineering SME.
• Drive adoption of Data Engineering best practices and standards.