Data Engineer

GSquared Group; Atlanta (GA); On-site; USD 120,000 - 160,000; Full time

Job summary

GSquared Group seeks a highly skilled Data Engineer to design, build, and support scalable data and streaming solutions in a cloud environment. You will work across the full data engineering lifecycle, gathering requirements and delivering lakehouse architectures and production pipelines using Databricks, Delta Lake, and Spark.

The ideal candidate has hands-on experience with Databricks, Delta Live Tables, Unity Catalog, Kafka, and AWS, plus strong SQL, data quality, observability, and

Qualifications

• Bachelor’s degree or equivalent in Information Systems, CS, EE, or related field.
• Hands-on experience with Databricks and modern data engineering practices.
• Strong experience with Delta Lake, Delta Live Tables, and Unity Catalog.
• Strong proficiency with Apache Spark, Spark SQL, and PySpark.
• Experience building streaming/data solutions with Kafka; Confluent Cloud a plus.
• Strong SQL and enterprise data transformation skills.
• Experience with AWS, especially S3 and related data integration patterns.
• Experience designing and operating production-grade data pipelines and scalable analytics solutions.
• Strong understanding of data quality, governance, observability, and maintainability.
• Excellent written and verbal communication across technical and business teams.

Responsibilities

• Own the architecture, design, development, and delivery of scalable data engineering and streaming solutions.
• Partner with stakeholders to translate requirements into durable data solutions.
• Design lakehouse architectures and production pipelines using Databricks and Delta Lake.
• Develop ingestion and transformation using Delta Live Tables, Spark, Spark SQL, and PySpark.
• Build real-time streaming data solutions using Kafka, including Confluent Cloud.
• Leverage AWS services (S3) to support enterprise data solutions.
• Develop complex SQL for data transformation, analysis, and integration.
• Apply software engineering practices to ensure scalability, maintainability, reliability, and security.
• Implement data quality, governance, observability, and monitoring practices.
• Troubleshoot production pipelines and streaming processes; resolve performance issues.
• Collaborate across teams to deliver data solutions for BI and analytics."

Skills

Databricks

Delta Lake

Unity Catalog

Apache Spark

Spark SQL

PySpark

SQL

AWS

Data quality

Observability

Governance

Data modeling

CI/CD

Streaming

Kafka

Education

Bachelor's degree in Information Systems/Computer Science/related field

Tools

Kafka

Confluent Cloud

Databricks

Delta Lake

Delta Live Tables

Unity Catalog

Apache Spark

Spark SQL

PySpark

S3/AWS services

Job description

We are seeking a highly skilled Data Engineer to design, build, and support scalable data and streaming solutions that transform enterprise data into trusted, governed datasets for business intelligence and advanced analytics.

The Data Engineer will work across the full data engineering lifecycle—from gathering and clarifying requirements with business stakeholders and data modelers to designing lakehouse architectures, developing production pipelines, and supporting solutions in a cloud environment. The ideal candidate brings strong hands‑on experience with Databricks, Delta Lake, Apache Spark, Kafka, AWS, and SQL, along with a strong engineering mindset focused on data quality, observability, scalability, and maintainability.

Responsibilities

• Own the architecture, design, development, and delivery of scalable data engineering and streaming solutions.
• Partner with business stakeholders, data modelers, and technical teams to understand requirements and translate business needs into durable data solutions.
• Design and implement lakehouse architectures and production‑grade data pipelines using Databricks and Delta Lake.
• Develop ingestion and transformation processes using Delta Live Tables (DLT), Apache Spark, Spark SQL, and PySpark.
• Build and support real‑time and streaming data solutions using Kafka, including managed platforms such as Confluent Cloud.
• Leverage AWS services, including S3 and related integration patterns, to support enterprise data solutions.
• Develop complex and efficient SQL for data transformation, analysis, and integration.
• Apply strong software engineering practices to ensure solutions are scalable, maintainable, reliable, and secure.
• Implement and maintain data quality, governance, monitoring, and observability practices.
• Troubleshoot production data pipelines and streaming processes and resolve performance and reliability issues.
• Collaborate across teams to define measurable outcomes and deliver data solutions that support BI and advanced analytics initiatives.
• Clearly communicate technical concepts, requirements, designs, and project status in both written and verbal formats.

Required Qualifications

• Bachelor’s degree required, preferably in Information Systems, Computer Science, Electrical Engineering, Computer Information Systems, or a related field.
• Advanced, hands‑on experience with Databricks and modern data engineering practices.
• Strong experience with Delta Lake, Delta Live Tables, and Unity Catalog.
• Strong proficiency with Apache Spark, including Spark SQL and PySpark.
• Hands‑on experience building streaming and event‑driven data solutions using Kafka; Confluent Cloud experience is a plus.
• Strong experience with SQL and enterprise data transformation.
• Experience with AWS, particularly S3 and related data integration services/patterns.
• Experience designing and operating production‑grade data pipelines and scalable analytics solutions.
• Strong understanding of data quality, governance, observability, and maintainability.
• Excellent written and verbal communication skills with the ability to work effectively across technical and business teams.

Preferred Qualifications

• Experience with Confluent Cloud and managed Kafka environments.
• Experience with enterprise lakehouse architecture and modern data platforms.
• Experience supporting BI, advanced analytics, or data science initiatives.
• Strong understanding of data modeling and data architecture concepts.
• Experience with CI/CD, source control, automated testing, and other modern engineering practices.