Senior Manager Data

Techsurge Private Limited; Karachi Division; On-site; PKR 3,000,000 - 6,000,000; Full time

Job summary

Techsurge Private Limited in Karachi is seeking a Senior Data Engineer to design and maintain scalable data architecture, build robust ETL/ELT pipelines, and develop data models and integration strategies aligned with business goals.

You will lead a team of data engineers, oversee data quality and governance, monitor pipelines, optimize performance, and evaluate new tools and frameworks to support banking/finance domain initiatives with Azure and on-premises environments.

Qualifications

• 12–15 years of experience in Big Data, Hadoop, Data Lake, Data Mesh architectures, Azure Cloud, MS SQL Server, Unix, and platform migrations.
• Must have led at least one large production go-live engagement.
• Strong knowledge of Cloud platform management (Azure/AWS) for DWH/Data Lake.
• Solid foundation in data management, data modeling, governance, and security.
• Proficient in SQL, Python, and JVM-based languages; familiar with Spark, Hive, and Kafka.

Responsibilities

• Design and maintain scalable data architecture, including data models and integration strategies.
• Develop and optimize ETL/ELT pipelines for data ingestion and transformation.
• Lead a team of data engineers; mentor, assign tasks, and drive collaboration.
• Ensure data quality, lineage, governance, and security across platforms.
• Monitor pipelines, troubleshoot issues, and implement fault-tolerant data solutions.
• Evaluate and integrate new data tools and frameworks to improve processes.
• Oversee data warehouse and data lake operations; manage performance optimization.
• Align data initiatives with banking/finance domain requirements and regulatory needs.

Skills

Big Data

Hadoop

Data Lake

Data Mesh

Azure Cloud

MS SQL Server

Unix

Apache Spark

PySpark

Hive

Azure Databricks

Azure Data Factory

SQL

Python

C#

Java

Kafka

Sqoop

ADLS

ADB

ADF

Tools

Hadoop Ecosystem Tools

Azure Data Factory

Azure Databricks

Kafka

Sqoop

ADLS

Job description

Job Description:

• Data Architecture Design: As a Senior Data Engineer, one of the primary responsibilities is to design and maintain the data architecture. This involves creating scalable, efficient, and robust data pipelines, data models, and data integration strategies.
• Business Focus Deliverables:
• Managed to build scalable & robust data visualization platform supported as per organization strategical tools & technology.
• Develops/maintains scalable Business Solutions for Wholesales Operations, Risk, Compliance/Client Experience (CX) Digitalization
• Understand clients' problems and decide how to cater to their needs, business problems.
• Proficient in requirement gathering, finalizing scope, consensus building across diverse stakeholders, formulating BRD's, concluding UAT, demos and trainings, tracking Agile Scrum & rituals.
• Team Leadership and Management: Leading a team of data engineers is a crucial aspect of this role. KRAs include managing team members (both FTE's and EC's), assigning tasks, mentoring junior engineers, and fostering a collaborative and productive work environment.
• Data Pipeline Development: Developing and optimizing data pipelines is essential to ensure smooth data flow from various sources to the data warehouse or data lake. This KRA involves implementing ETL/ELT processes, data ingestion, and transformation workflows.
• Data Quality and Governance: Ensuring the quality, integrity, lineage, and security of data is another vital aspect of a Lead Data Engineer's role. This involves implementing data governance policies, data monitoring mechanisms, and error handling procedures.
• Performance Optimization: Responsible for optimizing data processing/query performance. This may involve tuning database queries, optimizing data storage, leveraging distributed computing technologies.
• Technology Selection and Integration: Keeping up with the latest trends and advancements in data engineering technologies is crucial. Evaluating and integrating new tools and frameworks to enhance the data engineering processes can be a significant KRA.
• Monitoring and Troubleshooting: Monitoring new and existing data pipelines to identify and resolve issues is an essential responsibility. This KRA involves setting up monitoring systems and establishing procedures for quick troubleshooting and problem resolution.
• Scalability and Resilience: Ensuring that the data engineering infrastructure can scale with increasing data volume and handle failures gracefully is critical. This KRA involves designing for high availability and fault tolerance.

Required Experience and Skills:

• 12 – 15 Years of Experience in Big Data, Hadoop, Data Lake, Data Mesh Architectures, Azure Cloud, Microsoft SQL Server, Unix, and platform migrations, Apache Spark, Pyspark, Hive, Azure data bricks, ADF.
• Should have managed successful execution of minimum 1large engagement into production go live.
• Must haveknowledge of Platform Management of Cloud, DWH, Azure and Data Lake.
• Good in Data Management Fundamentals and Data Architect, Modelling, Governance and Data security.
• Strong in writing & tuning the Data Ingestion jobs, Spark applications, python, Hive, Azure Databricks, Azure Data Factory and Azure SQL.
• Proficient in SQL, Python, C#, Java, or another JVM-based language.
• Proficient in Spark, Hive, Sqoop, ADLS, ADB, ADF and Kafka.
• Public/Private cloud experience in AWS and/or Azure.
• Good knowledge of RDBMS/NoSQL database design and best practices.
• Experience in DWH to Data Lake offloading.
• Proficient in Data Modeling and Data Governance concepts.
• Good Domain Knowledge in Banking/Finance area.