Cloud Data Engineer - Pipelines, Warehouses & AI Readiness

Compunnel Inc.; Atlanta (GA); Hybrid; USD 120,000 - 180,000; Full time

Job summary

Compunnel, Inc. seeks a skilled Data Engineer to design, develop, and optimize scalable pipelines and platforms supporting analytics, AI/ML, and healthcare operations.

The role emphasizes cloud-based ETL/ELT development, big data tech, data quality, and governance while collaborating with analysts, architects, and data scientists to meet enterprise data requirements.

Qualifications

• Experience in data engineering and building scalable data pipelines.
• Experience with ETL/ELT design and development.
• Proficiency in data warehousing concepts and data modeling.
• Strong understanding of data lake architectures.
• Experience with cloud platforms (AWS or Azure).
• Proficiency with SQL databases (SQL Server, Oracle, PostgreSQL, Snowflake).
• Proficiency with Python, SQL, PySpark, and shell scripting.
• Experience with Spark/Hadoop ecosystem.
• Familiarity with DevOps and CI/CD practices (Git, Jenkins, Azure DevOps).

Responsibilities

• Design, develop, and maintain scalable ETL/ELT pipelines for structured and unstructured data.
• Build and optimize data ingestion frameworks from multiple source systems, APIs, databases, and cloud platforms.
• Develop and maintain enterprise data models, data warehouses, and data lakes.
• Ensure high levels of data quality, integrity, security, and governance across data platforms.
• Collaborate with business analysts, architects, data scientists, and application teams to understand data requirements.
• Implement data transformation, validation, reconciliation, and monitoring processes.
• Optimize performance of large-scale data pipelines and database workloads.
• Support analytics, reporting, AI/ML, and advanced data science initiatives through reliable data delivery.
• Implement metadata management, lineage tracking, and data cataloging solutions.
• Troubleshoot production issues and perform root cause analysis for data-related incidents.
• Participate in code reviews, architecture discussions, and technical design sessions.
• Ensure adherence to healthcare regulatory and compliance requirements, including HIPAA and data privacy standards.
• Work with distributed teams across multiple locations and support enterprise-scale data initiatives.

Skills

Data Engineering

Data Pipeline Development

ETL/ELT Design and Development

Data Warehousing Concepts

Data Modeling

Data Lake Architecture

Data Quality and Data Governance

Cloud Technologies (AWS/Azure)

Databases (SQL Server, Oracle, Postgre

Python

SQL

PySpark

Shell Scripting

Big Data Technologies (Spark, Hadoop)

DevOps & CI/CD (Git, Jenkins/Azure Dev

Education

Bachelor's or Master's degree in Computer Science, Information Technology, Data Engineering, or related field

Tools

AWS

Azure

Databricks

Snowflake

Job description

plaintext
JOB SUMMARY
We are seeking a skilled Data Engineer to design, develop, and optimize scalable data pipelines and data platforms that support enterprise analytics, reporting, AI/ML initiatives, and healthcare business operations. The ideal candidate will have strong expertise in cloud-based data engineering, ETL/ELT development, big data technologies, and data quality practices, with experience working in healthcare environments.

Key Responsibilities
Design, develop, and maintain scalable ETL/ELT pipelines for structured and unstructured data.
Build and optimize data ingestion frameworks from multiple source systems, APIs, databases, and cloud platforms.
Develop and maintain enterprise data models, data warehouses, and data lakes.
Ensure high levels of data quality, integrity, security, and governance across data platforms.
Collaborate with business analysts, architects, data scientists, and application teams to understand data requirements.
Implement data transformation, validation, reconciliation, and monitoring processes.
Optimize performance of large-scale data pipelines and database workloads.
Support analytics, reporting, AI/ML, and advanced data science initiatives through reliable data delivery.
Implement metadata management, lineage tracking, and data cataloging solutions.
Troubleshoot production issues and perform root cause analysis for data-related incidents.
Participate in code reviews, architecture discussions, and technical design sessions.
Ensure adherence to healthcare regulatory and compliance requirements, including HIPAA and data privacy standards.
Work with distributed teams across multiple locations and support enterprise-scale data initiatives.

Required Qualifications
Data Engineering
Data Pipeline Development
ETL/ELT Design and Development
Data Warehousing Concepts
Data Modeling (Dimensional & Relational)
Data Lake Architecture
Data Quality and Data Governance
Cloud Technologies AWS (Glue, EMR, Lambda, S3, Redshift, Athena) or Azure (ADF, Data Lake, Synapse, Databricks)
Databases SQL Server Oracle PostgreSQL Snowflake
Programming Python SQL PySpark Shell Scripting
Big Data Technologies Apache Spark Hadoop Ecosystem
DevOps & CI/CD Git Jenkins/Azure DevOps CI/CD Pipeline Implementation

Preferred Qualifications
Experience in Healthcare, Health Insurance, or Life Sciences domain.
Exposure to AI/ML data pipelines and MLOps frameworks.
Knowledge of healthcare data standards such as HL7, FHIR, ICD, and CPT is preferred.
Experience with Snowflake, Databricks, or modern cloud-native data platforms.
Familiarity with Agile/Scrum methodology.
Education Bachelor's or Master's degree in Computer Science, Information Technology, Data Engineering, or related field.

Certifications
None