ETL SQL / Enterprise Data Framework Developer

IntraEdge; Hyderabad; On-site; INR 1,400,000 - 2,200,000; Full time

Benefits offered by this job

 

Job summary

IntraEdge is seeking an experienced ETL SQL / Data Engineer to design and maintain data pipelines and EDF jobs. The role focuses on SQL development, stored procedures, views, and scheduling with Autosys.

You'll work with EDF modules like DCAF, ensuring data quality and validation across flat-file and REST API ingestion. Experience with Data Lake and Snowflake is valued, along with problem-solving in enterprise-scale pipelines.

Qualifications

• 3–6 years of experience in Data Engineering / ETL development.
• Strong SQL development, data pipeline design, and performance optimization.
• Hands-on with EDF and enterprise data frameworks; experience with DCAF is a plus.
• Experience with Autosys scheduling and job monitoring.

Responsibilities

• Develop, optimize, and maintain ETL SQL code for enterprise data pipelines.
• Create and manage EDF jobs and modules (D2P, DCAF, GPR).
• Handle data extraction from flat files and REST APIs; ensure data quality and validation.
• Load data into target platforms like Data Lake and Snowflake and perform post-load checks.
• Configure Autosys schedules, dependencies, and calendars; monitor jobs and troubleshoot failures.
• Support testing, deployment, and production operations; collaborate with cross-functional teams.

Skills

SQL Development

ETL Pipeline Development

Stored Procedures

Views

DBeaver

EDF

Autosys Scheduling

REST APIs

Flat Files

Data Validation

Snowflake Data Lake

Tools

DBeaver

Autosys

EDF

Job description

Experience: 3–6 Years

Employment Type: Full-Time

Job Summary

We are seeking an experienced ETL SQL / Data Engineer with strong expertise in SQL development, ETL data pipelines, Stored Procedures, Views, Autosys scheduling, and Enterprise Data Framework (EDF).

The candidate will be responsible for developing and maintaining data pipelines that extract data from multiple source systems, process and validate the data through EDF modules, and load it into target platforms such as Data Lake and Snowflake.

The ideal candidate should have strong SQL coding skills, hands‑on experience with DBeaver, and a good understanding of data integration using flat files and REST APIs. Experience working with enterprise‑scale data pipelines and scheduling frameworks is highly desirable.

Key Responsibilities

ETL & SQL Development

• Develop, maintain, and optimize ETL SQL code for enterprise data pipelines.
• Write complex SQL queries for data extraction, transformation, validation, reconciliation, and loading.
• Develop and maintain:
• Stored Procedures
• Views
• SQL scripts
• Data validation and reconciliation queries
• Analyze source data and determine appropriate transformation and mapping logic.
• Optimize SQL queries for performance and scalability.
• Perform data quality checks and troubleshoot data discrepancies.
• Ensure data pipelines meet defined business and technical requirements.

Enterprise Data Framework (EDF)

• Develop new Enterprise Data Framework (EDF) jobs to support enterprise data processing requirements.
• Work extensively with EDF to design, build, test, deploy, and support data pipelines.
• Understand and utilize EDF modules including:
• Data Control and Anomaly Detection Framework (DCAF)
• Build EDF jobs to extract data from source systems and process the data through the appropriate EDF modules.
• Configure data processing, validation, anomaly detection, and reconciliation rules.
• Ensure successful movement of data across different stages of the pipeline.
• Troubleshoot EDF job failures and data processing issues.
• Develop reusable and maintainable EDF components wherever possible.

Data Extraction & Integration

• Build data extraction processes from multiple source systems.
• Work with flat files as source data, including file‑based ingestion and processing.
• Understand API request/response structures and troubleshoot API‑related data ingestion issues.
• Validate incoming data for completeness, accuracy, format, and quality.
• Handle different data formats and transformation requirements.
• Ensure reliable movement of data from source systems through the EDF pipeline.

Data Processing & Transformation

• Process extracted data through EDF's D2P, DCAF, and GPR modules.
• Implement transformation and business rules required for downstream processing.
• Develop data validation and anomaly detection mechanisms.
• Implement reconciliation logic to compare source and target datasets.
• Investigate data quality issues and work with upstream teams to resolve them.
• Ensure data is accurately transformed before loading into target environments.

Target Data Platforms

• Build and support pipelines that load processed data into target environments such as:
• Data Lake
• Snowflake
• Validate successful data loading and perform post‑load data quality checks.
• Reconcile source and target records to ensure completeness and accuracy.
• Troubleshoot data load failures and performance issues.
• Work with Data Engineering and platform teams to resolve target‑system issues.

Autosys Scheduling

• Develop, configure, and maintain Autosys jobs for data pipeline scheduling.
• Define job dependencies, calendars, conditions, and execution sequences.
• Monitor scheduled ETL and EDF jobs.
• Troubleshoot failed or delayed jobs.
• Manage dependencies between upstream and downstream data processes.
• Support production scheduling and batch‑processing requirements.

Data Quality & Reconciliation

• Implement data quality checks throughout the ETL lifecycle.
• Utilize DCAF capabilities for anomaly detection and data control.
• Investigate data mismatches between source and target environments.
• Identify data anomalies and work with relevant teams to resolve them.
• Ensure data pipelines meet defined quality and completeness standards.

Development, Testing & Deployment

• Participate in the complete development lifecycle:
• Requirement analysis
• Development
• Unit testing
• Integration testing
• Deployment
• Production support
• Develop test cases for SQL, ETL, and EDF jobs.
• Perform unit and integration testing of data pipelines.
• Validate data transformation and reconciliation results.
• Support UAT and production deployments.
• Troubleshoot defects and implement fixes within agreed timelines.

Production Support

• Monitor ETL and EDF pipelines and proactively identify failures.
• Analyze job logs and error messages to determine root causes.
• Resolve production issues related to SQL, ETL, Autosys, APIs, flat files, and EDF.
• Coordinate with application, infrastructure, source‑system, and database teams.
• Participate in incident and problem management activities.

Required Technical Skills

SQL

• Strong knowledge of:
• Stored Procedures
• Views
• Joins
• Subqueries
• Window Functions
• Aggregations
• Data validation
• Ability to optimize SQL queries and troubleshoot performance issues.

DBeaver

• Hands‑on experience using DBeaver for SQL development and database analysis.
• Ability to analyze database objects, execute queries, troubleshoot SQL issues, and validate data.
• Experience developing or supporting Enterprise Data Framework (EDF) jobs is highly preferred.
• Understanding of:
• DCAF – Data Control and Anomaly Detection Framework
• Ability to develop new EDF jobs and troubleshoot existing pipelines.

Data Sources

Strong understanding of data extraction from:

• Flat files
• REST APIs
• Relational databases

Data Targets

Experience working with:

• Data Lake
• Snowflake
• Cloud‑based data platforms

Scheduling

• Hands‑on experience with Autosys.
• Knowledge of job scheduling, dependencies, calendars, triggers, and batch processing.

Preferred Skills

• Experience with Snowflake.
• Experience working with cloud‑based Data Lakes.
• Knowledge of Python or shell scripting.
• Experience with REST API integration.
• Understanding of JSON/XML data formats.
• Experience with data quality and data governance concepts.
• Knowledge of CI/CD and source‑control tools such as Git.
• Exposure to Agile/Scrum development methodologies.
• Experience working with enterprise‑scale data platforms.

Key Responsibilities at a Glance

The selected candidate will primarily be responsible for:

• Developing ETL SQL code, Stored Procedures, and Views.
• Building new EDF jobs for enterprise data pipelines.
• Extracting data from flat files and REST APIs.
• Processing data through D2P, DCAF, and GPR modules.
• Implementing data validation, anomaly detection, and reconciliation.
• Loading processed data into Data Lake or Snowflake.
• Creating and managing Autosys schedules and dependencies.
• Performing data quality checks and troubleshooting pipeline issues.
• Supporting testing, deployment, and production operations.
• Working with cross‑functional teams to ensure reliable and accurate data delivery.

Candidate Profile

• 3–6 years of experience in Data Engineering, ETL Development, SQL Development, or Data Integration.
• Strong SQL programming and data analysis skills.
• Hands‑on experience with DBeaver.
• Strong understanding of ETL concepts and data pipeline development.
• Experience with Autosys scheduling.
• Experience with flat‑file and REST API‑based data ingestion is mandatory.
• Experience with EDF or a similar enterprise data pipeline framework is highly preferred.
• Strong understanding of data quality, reconciliation, and transformation.
• Good troubleshooting and analytical skills.
• Ability to work independently as well as collaboratively with distributed technical teams.