開放取用學術資源的自動化擷取系統實作：以臺灣人文社會領域期刊資料為例

Abstract

本研究旨在建置一套可穩定運作且具高度擴展性的自動化學術資源擷取系統，針對臺灣人文社會領域開放取用期刊進行抓取。現行國內引文索引資料庫多仰賴人工建檔與維護，導致資料更新與整合流程費時費力；而開放取用平台則受限於期刊端主動上架與維運意願，造成資料時效與涵蓋範圍不足，進而影響學術資源的可用性與知識庫建構的穩定性。為此，本研究設計並實作「Social and Theoretical Academic Repository（STAR）」系統，結合 Scrapy 爬蟲框架與 Docker 容器化部署技術，整合 MySQL、Redis、Django、Playwright、FTP 等模組，建立排程式爬取、結構化檔案儲存與網頁式管理操作的自動化平台。系統具備網頁式管理介面，支援管理者透過 Django 後台調整排程或即時執行爬蟲任務，亦提供 FTP 批次下載功能供使用者取得期刊全文檔案。系統完成部署後共建置 46 支期刊爬蟲模組，成功擷取 17,865 篇 PDF 文章檔案，總容量達 79 GB。比較首次與後續爬取平均耗時，整體處理效率提升 73.4 %，顯示系統具備長期穩定運作與低維運負擔的特性。本研究驗證了以模組化容器架構整合開源爬蟲技術，能有效支援多網站資料擷取與期刊資料彙整之需求，並為後續文本生成、語意比對與知識問答等應用場景，提供可重複使用之期刊資料擷取基礎。未來可進一步結合語意嵌入與文本分析工具，拓展資料加值應用場景。

This study aims to develop a stable and highly scalable automated system for extracting open-access academic resources, specifically targeting journals in the humanities and social sciences in Taiwan. Existing domestic citation index databases rely heavily on manual curation and maintenance, resulting in time-consuming and labor-intensive update and integration processes. In addition, open access platforms are constrained by the willingness of journal publishers to actively upload and maintain content, leading to issues of timeliness and coverage, which ultimately affect the usability of academic resources and the stability of knowledge base construction.To address these challenges, this study designs and implements the “Social and Theoretical Academic Repository” (STAR) system. The system integrates the Scrapy web crawling framework with Docker-based container deployment, combining MySQL, Redis, Django, Playwright, and FTP modules to create an automated platform featuring scheduled crawling, structured file storage, and web-based administrative control.The system provides a web interface that allows administrators to adjust schedules or execute crawlers in real time via the Django backend. It also offers FTP-based batch downloading for users to access full-text journal PDFs. Upon deployment, 46 crawler modules were implemented, successfully harvesting 17,865 PDF articles with a total data volume of 79 GB. A comparison of initial and subsequent crawls showed a 73.4 % improvement in processing efficiency, demonstrating the system's long-term stability and low maintenance cost. This study confirms that integrating open-source crawling technologies within a modular container architecture effectively supports multi-site data extraction and journal metadata aggregation. The system also provides a reusable infrastructure for future applications in text generation, semantic similarity analysis, and knowledge-based question answering. Further development may incorporate semantic embedding and text analysis techniques to enhance value-added applications.