健康心理因素文本自動分類之研究

Abstract

心理學的研究對象通常非常複雜，需要長期追蹤和研究。傳統的研究方法需要人工標記和評分，這不僅費時費力，還容易出現主觀性和一致性問題。目前大多數研究透過社群平台來找到研究對象。因此本研究希望透過社群平台找到研究資料，並利用機器以自動化的方式更有效的進行心理學研究。本研究要將心理健康方面的文本用人工智慧的技術，將其自動分類到5個面向中的11個指標，每個指標都有5個分數，並且期望在有限的人工標記的訓練資料下（每個類至少60筆資料），機器預測的準確度要能達到0.8以上（人工標記一致性平均分數為0.8011），以Macro F1為主要判斷標準。使用的技術包括機器學習、BERT、SetFit、GPT-3、GPT-4。就本研究的結果而言，機器學習與BERT雖然執行的時間成本低，但成效在各指標都無法達到理想的0.8。GPT-4也許因為是使用prompt的方式進行實驗，要它處理的任務太過於複雜，準確度無法像用訓練的方式來的好，所以也都沒有達到目標。GPT-3與SetFit的成效在多數指標上都有不錯的表現，GPT-3有5個指標達到目標，SetFit更是有7個指標達到目標，兩個指標只差1到2個百分點達到目標。 考量到GPT-3的執行時間成本很重（主要是1次request只能預測1筆資料），而SetFit只有訓練時間成本重而已，預測的速度是非常快速的，所以選用SetFit用於心理健康文本的自動分類是一個準確度高、預測時間成本低的方法。Subjects of psychological studies are often intricate, requiring long-term tracking and research. Traditional research methods involve manual labeling and scoring, which is not only time-consuming but also prone to subjectivity and consistency issues. Presently, most studies leverage social media platforms to find subjects for research. Therefore, this study aims to acquire research data through social media platforms and utilize machine-driven automation for more efficient psychological research.This research aims to employ artificial intelligence techniques in the domain of mental health to automatically categorize text into 11 indicators across 5 aspects, with each indicator having 5 scores. The objective is to achieve a machine prediction accuracy of 0.8 or higher with limited manually labeled data (approximately 60 data points per class), with human-labeled consistency averaging a score of 0.8011, using Macro F1 as the primary evaluation metric.Techniques employed include machine learning, BERT, SetFit, GPT-3, and GPT-4. In terms of the results of this study, while machine learning and BERT incur low execution time costs, their effectiveness falls short of the desired 0.8 across all indicators. GPT-4, perhaps due to the complexity of the tasks it is prompted to handle, fails to achieve the desired accuracy, as it cannot perform as effectively as in a trained mode. GPT-3 and SetFit show promising performance across multiple indicators, with GPT-3 reaching the target in 5 indicators and SetFit achieving the target in 7 indicators, with only a 1 to 2 percentage point difference in two indicators to reach the target.Considering the significant execution time cost of GPT-3 (mainly due to predicting one data point per request) and the primary training time cost of SetFit, the prediction speed of SetFit is considerably rapid. Therefore, choosing SetFit for automatic classification of psychological text proves to be a high-accuracy, low prediction time cost method.