Senior Data Engineer: Web Scraping & SQL Pipelines

Market Calculus Türkiye; Çankaya; On-site; TRY 360,000 - 600,000; Full time

Job summary

Market Calculus Türkiye is seeking a Senior Data Engineer to join its R&D Center in Turkey. You will lead data scraping, integration, and the development of ETL pipelines, focusing on reliable and scalable data processes.

The role requires strong SQL skills, hands-on experience with PostgreSQL, and proficiency in web scraping, REST APIs, JSON, and HTML data. Collaboration across teams and database optimization are key aspects.

Qualifications

• 5+ years of experience in Data Engineering/ETL.
• Strong to advanced SQL skills.
• Experience with PostgreSQL or a similar relational database.
• Hands-on experience with web scraping.
• Experience processing REST APIs, JSON, and HTML data.
• Experience with Git.
• Comfortable working in Linux environments.

Responsibilities

• Collect data from various web sources and develop/improve scraping processes using Scrapy, Selenium, Requests, and similar tools.
• Monitor scraping processes and resolve issues as they arise.
• Compare and integrate data from new sources with existing datasets.
• Perform data matching, duplicate detection, and data quality checks.
• Develop, maintain, and monitor ETL pipelines.
• Improve data models and SQL queries.
• Design scalable data processing solutions leveraging database capabilities.
• Optimize data processing for data volume, query performance, and scalability.

Skills

SQL

PostgreSQL

Web scraping

REST APIs

JSON

HTML

Git

Linux

Tools

Scrapy

Selenium

Requests

Docker

Elasticsearch/OpenSearch

Job description

Market Calculus Türkiye is seeking a Senior Data Engineer to join its R&D Center in Turkey. You will lead data scraping, integration, and the development of ETL pipelines, focusing on reliable and scalable data processes.

The role requires strong SQL skills, hands-on experience with PostgreSQL, and proficiency in web scraping, REST APIs, JSON, and HTML data. Collaboration across teams and database optimization are key aspects.