Lead Data Engineer

TechDigital Group; Warsaw (IN); On-site; USD 80,000 - 120,000; Full time

Job summary

TechDigital Group is seeking a data engineer based in Warsaw, Indiana, to design and implement state-of-the-art data engineering architectures. The role involves advanced ETL/ELT pipeline development, handling both batch and real-time streaming data, and ensuring scalability across data lakes and warehouses. Candidates should have strong expertise in cloud services (Azure, AWS, GCP), data warehouse technologies (Snowflake, Synapse, BigQuery, Redshift), and demonstrate a commitment to data quality and operational excellence.

Responsibilities

• Design end to end data engineering architectures.
• Define scalable solutions for data lakes, data warehouses, and streaming systems.
• Lead development of ETL / ELT pipelines and optimize pipelines for performance.
• Establish data quality standards and perform root cause analysis for incidents.

Skills

ETL / ELT pipeline design

Batch data processing

Near real-time / streaming data

Programming skills in Python or Scala or Java

Performance optimization

Data Warehouses (Snowflake, Synapse, BigQuery, Redshift)

Azure, AWS, or GCP

Job description

Job Details:

• Advanced expertise in ETL / ELT pipeline design
• Handling:
  • Batch data processing
  • Near real time / streaming data
• Experience with structured and semi structured data
• Strong knowledge of:
  • Incremental loading
  • CDC (Change Data Capture)
• Pipeline orchestration and dependency management
• Strong programming skills in Python or Scala or Java (nice to have)
• Performance optimization for large scale data processing
• Solid understanding of:
  • Dimensional modeling (Star / Snowflake)
  • Normalized and denormalized models
• Strong experience on: Azure, AWS, or GCP
• Hands on with Data Warehouses (Snowflake, Synapse, BigQuery, Redshift)

Data Architecture & Solution Design

• Design end to end data engineering architectures
• Define scalable solutions for:
  • Data lakes / lakehouse
  • Data warehouses
  • Streaming and real time systems
• Ensure alignment with enterprise architecture, security, and compliance standards
• Review and approve technical designs

Data Pipeline Development & Management

• Lead development of ETL / ELT pipelines
• Handle:
  • Batch and real time ingestion
  • Structured and semi structured data
• Optimize pipelines for performance, reliability, and cost
• Manage schema evolution and data dependencies

Data Quality, Reliability & Operations

• Establish data quality standards and validation rules
• Implement monitoring, alerting, and observability
• Perform root cause analysis for data incidents
• Drive operational excellence and stability

DevOps / DataOps Enablement

• Build CI/CD pipelines for data workloads
• Automate testing, deployment, and rollback
• Improve reliability through automation