Data Engineer: Scalable Lakehouse & Real-Time Pipelines

GSquared Group; Atlanta (GA); Presencial; USD 120.000 - 160.000; Jornada completa

Descripción de la vacante

GSquared Group seeks a highly skilled Data Engineer to design, build, and support scalable data and streaming solutions in a cloud environment. You will work across the full data engineering lifecycle, gathering requirements and delivering lakehouse architectures and production pipelines using Databricks, Delta Lake, and Spark.

The ideal candidate has hands-on experience with Databricks, Delta Live Tables, Unity Catalog, Kafka, and AWS, plus strong SQL, data quality, observability, and

Formación

• Bachelor’s degree or equivalent in Information Systems, CS, EE, or related field.
• Hands-on experience with Databricks and modern data engineering practices.
• Strong experience with Delta Lake, Delta Live Tables, and Unity Catalog.
• Strong proficiency with Apache Spark, Spark SQL, and PySpark.
• Experience building streaming/data solutions with Kafka; Confluent Cloud a plus.
• Strong SQL and enterprise data transformation skills.
• Experience with AWS, especially S3 and related data integration patterns.
• Experience designing and operating production-grade data pipelines and scalable analytics solutions.
• Strong understanding of data quality, governance, observability, and maintainability.
• Excellent written and verbal communication across technical and business teams.

Responsabilidades

• Own the architecture, design, development, and delivery of scalable data engineering and streaming solutions.
• Partner with stakeholders to translate requirements into durable data solutions.
• Design lakehouse architectures and production pipelines using Databricks and Delta Lake.
• Develop ingestion and transformation using Delta Live Tables, Spark, Spark SQL, and PySpark.
• Build real-time streaming data solutions using Kafka, including Confluent Cloud.
• Leverage AWS services (S3) to support enterprise data solutions.
• Develop complex SQL for data transformation, analysis, and integration.
• Apply software engineering practices to ensure scalability, maintainability, reliability, and security.
• Implement data quality, governance, observability, and monitoring practices.
• Troubleshoot production pipelines and streaming processes; resolve performance issues.
• Collaborate across teams to deliver data solutions for BI and analytics."

Conocimientos

Databricks

Delta Lake

Unity Catalog

Apache Spark

Spark SQL

PySpark

SQL

AWS

Data quality

Observability

Governance

Data modeling

CI/CD

Streaming

Kafka

Educación

Bachelor's degree in Information Systems/Computer Science/related field

Herramientas

Kafka

Confluent Cloud

Databricks

Delta Lake

Delta Live Tables

Unity Catalog

Apache Spark

Spark SQL

PySpark

S3/AWS services

Descripción del empleo

GSquared Group seeks a highly skilled Data Engineer to design, build, and support scalable data and streaming solutions in a cloud environment. You will work across the full data engineering lifecycle, gathering requirements and delivering lakehouse architectures and production pipelines using Databricks, Delta Lake, and Spark.

The ideal candidate has hands-on experience with Databricks, Delta Live Tables, Unity Catalog, Kafka, and AWS, plus strong SQL, data quality, observability, and