Principal Data Engineer

Tkxel LLC; Lahore; On-site; PKR 27,777,778 - 38,888,889; Full time

Job summary

Tkxel LLC, a software development company based in Reston, Virginia, is seeking a data engineering specialist to build scalable data pipelines using Azure and Spark ecosystems. You will design ETL processes, manage data lakes, and ensure data quality and compliance across enterprise systems.

The role focuses on data integration, governance, and performance tuning to support business insights and analytics with modern data platforms.

Qualifications

• Azure Data Lake or Apache Delta Lake
• Apache Spark or Databricks
• Knowledge of implementing PySpark

Responsibilities

• Data pipeline development using Microsoft Fabric suite and related tech
• Data availability assessment and robust data collection/validation processes
• ETL development to central data repository with accuracy and accessibility
• Data Lake storage design with Azure OneLake for efficient access
• Data integration across business systems into a unified platform
• Implement data governance and quality controls including validation and cleansing
• Monitor and optimise pipeline and database performance
• Ensure data security and compliance with regulations and policies

Tools

Azure Data Lake

Apache Spark

Databricks

PySpark

Job description

Tkxel is a leading softwaredevelopment company located in Reston, Virginia. We are committedto develop innovative software solutions for leading enterprisesin the world, helping them grow their businesses using latesttechnology solutions.

Job Description

• Data Pipeline Development: Builds understanding of the data needs of the clinet and designs, constructs, installs, tests, and maintains highly scalable data management systems using Microsoft Fabric suite (Azure Data Factory, Azure Synapse Analytics, etc.) and other relevant technologies that can efficiently and effectively meet those needs
  
• Perform data availability assessment and implement robust processes to ensure the timely identification, collection, and validation of relevant data sets required for the client
  
• ETL Processes: Develops ETL processes to extract data from various sources, transform the data according to business rules, and load it into a centralised data repository, ensuring data accuracy and availability.
  
• Data Lake: Implements and manages data storage solutions using Azure OneLake and ensures optimal data storage architecture for ease of access and analysis.
  
• Data Integration: Integrates data from various business systems into a unified data platform, enabling a consolidated view of information across the organization.
  
• Data Quality and Governance: Ensures data accuracy and quality by implementing data governance and quality control measures, including data validation and cleansing.
  
• Performance Optimisation: Monitors, tunes, and reports on the performance of data pipelines and databases to ensure they meet the functional and performance requirements.
  
• Security and Compliance: Implements security measures to protect data integrity and compliance with data protection regulations and company policies.
  

Requirements

Skills:

• Azure Data Lake orApache Delta Lake
  
• Apache Spark or Databricks,
  
• Knowledge of implementing Apache Spark using pyspark