Data Tech Lead

Coforge; Fort Mill (SC); On-site; USD 140,000 - 190,000; Full time

Job summary

Coforge in Fort Mill, SC seeks a Data Tech Lead to design, build, and govern scalable data platforms on AWS. You will guide Lakehouse strategies, batch and streaming pipelines, and data quality controls.

You will optimize performance, implement metadata catalogs, and lead Airflow/MWAA orchestration, IAM governance, and cost-efficiency across the data estate.

Qualifications

• Experience designing scalable enterprise data platforms on AWS.

Responsibilities

• Lead the design, development, and support of scalable enterprise data platforms using AWS cloud-native services.
• Guide Data Lake and Lakehouse implementation using Amazon S3, AWS Glue, Lake Formation, Athena, Postgres, Apache Iceberg, Delta Lake, and Parquet.
• Develop and review batch, incremental, and CDC pipelines from databases, APIs, files, and streaming platforms.
• Build and optimize ETL/ELT workflows using AWS Glue, Spark, PySpark, Python, SQL, and Lambda.
• Create reusable ingestion, transformation, validation, reconciliation, and monitoring frameworks.
• Implement real-time processing with Kafka/Kinesis/SNS/SQS/EventBridge.
• Lead workflow orchestration with Airflow/MWAA/Step Functions.
• Implement metadata, catalog, lineage, and discovery with AWS Glue Data Catalog and Lake Formation.
• Improve performance and cost efficiency via partitioning, tuning, and storage optimizations.
• Enforce data quality controls and secure access with IAM, KMS, and Lake Formation permissions.

Skills

Data architecture

Data governance

Leadership

Mentoring

Problem solving

Tools

AWS

S3

AWS Glue

Lake Formation

Athena

Postgres

Apache Iceberg

Delta Lake

Parquet

Kafka

Kinesis

Airflow

MWAA

AWS Step Functions

IAM

Job description

Location: Fort Mill SC

We at Coforge are hiring Data Tech Lead with the following skill sets.

Job Description

• Lead the design, development, and support of scalable enterprise data platforms using AWS cloud-native services and modern data engineering technologies.
• Guide Data Lake and Lakehouse implementation using Amazon S3, AWS Glue, Lake Formation, Athena, Postgres, Apache Iceberg, Delta Lake, and Parquet.
• Develop and review batch, incremental, and Change Data Capture pipelines for data from databases, APIs, files, enterprise applications, and streaming platforms.
• Build and optimize ETL and ELT workflows using AWS Glue, Apache Spark, PySpark, Python, SQL, Lambda, and distributed processing frameworks.
• Create reusable frameworks for ingestion, transformation, validation, reconciliation, exception handling, error recovery, and operational monitoring.
• Implement real-time and near-real-time processing using Kafka, Amazon MSK, Amazon Kinesis, Lambda, SNS, SQS, and EventBridge.
• Lead workflow orchestration using Apache Airflow, Amazon MWAA, AWS Step Functions, and event-based scheduling.
• Implement metadata, schema, catalog, lineage, and discovery capabilities using AWS Glue Data Catalog and Lake Formation.
• Improve performance and cost efficiency through partitioning, compaction, retention, lifecycle management, storage optimization, Spark tuning, and query optimization.
• Enforce data quality controls covering completeness, accuracy, consistency, integrity, uniqueness, and source-to-target reconciliation.
• Implement secure and governed access using IAM, KMS, S3 policies, Lake Formation permissions, encryption, and fine-grained controls.