語法自動檢測工具比較及其精準度評估之研究

Abstract

許多 EFL 學生將寫作視為巨大的挑戰，因此開始尋求線上寫作輔助系統的輔助。隨著科技的發展和對於線上寫作輔助系統日益增加的關注，逐漸推動了各種線上語法檢查工具的發展和普及。近年來，許多語法檢查器不僅可以直接在網路瀏覽器上免費使用，有些甚至還提供付費升級的版本。由於這些語法檢查工具迅速激增並且研究有限，因此本研究旨在以準確率、錯誤建議的發生率和誤報的次數來評估和比較五個語法自動檢測工具，即Grammarly、LanguageTool、Ginger、Zoho Writer和NTHU Checker，以確定這些工具是不是足夠準確成為老師和學生可以作為寫作輔助的工具。檢測方法為從 Common Mistakes in English 書中檢索出20個常見錯誤並用於測試這五個語法檢查工具。總共檢測 560 個錯誤的句子，並記錄了這些工具識別每種類型的語法錯誤的能力。結果顯示，所有五款語法檢查器的準確率均較低，只有Grammarly的準確率超過50%，其次是Zoho Writer，然後是NTHU Checker，LanguageTool和Ginger 是準確率最低的語法檢查器。儘管總體覆蓋率較低，但所有五種語法檢查器都顯示出極少的誤報而且錯誤建議的發生率也很低，這表明語法檢查器的最大缺陷是傾向於忽略錯誤而不是提供不准確的糾正。在整體性能方面，Grammarly獲得最高的準確性，同時在避免錯誤建議的發生和誤報方面做得很好。語法檢測研究中較少被檢測的Zoho Writer，也表現得非常好，擁有第二高準確率，並且比Grammarly更好地避免不準確的更正和誤報，更重要的是這個工具在網路上可以免費取得。相比之下，Ginger和LanguageTool不僅準確率低，錯誤建議的發生率也很高。另一方面，NTHU Checker雖然表現出平均性能，然而與其他語法檢查器相比，生成的改正和誤報相對更不準確。研究結果還揭示了根據錯誤類型，準確率存在相當大的差異。據觀察，語法檢查器有擅長處理某些錯誤類型。更容易檢測和糾正的語法錯誤包括介係詞、不定詞、時態、單複數混淆和副詞位置錯誤。相比之下，語法檢查器在檢測英語中的慣用語、詞序錯誤和需要更多上下文資訊才能檢測和更正的錯誤方面存在局限性。最後，本研究根據研究結果提出了一些老師可以如何在課堂中運用語法檢查器來輔助教學的建議和方法，包括文法教學和寫作教學上的運用，幫助老師和學生可以更輕鬆並正確的使用語法檢查器。

Many EFL students see writing as a great challenge and therefore start to seek online writing support for help. Increased attention to writing assistants and technological advancement have driven the development and popularity of various grammar checking tools. Many grammar checkers now can be accessed directly through a web browser, with free and premium versions available. The question is how efficient these grammar checkers are. Therefore, this study examined five grammar checkers' performance on error correction ability. The aim is to compare the reliability of different grammar checkers (Grammarly, LanguageTool, Ginger, Zoho Writer, and NTHU Checker) in terms of overall accuracy, appropriacy of suggested corrections, and avoidance of false alarms. A total of 560 ungrammatical sentences under twenty error types retrieved from Common Mistakes in English were used to test these grammar checkers. The results showed that all five grammar checkers suffered from poor accuracy rates and that Grammarly was the only one whose accuracy rate exceeded 50%, followed by Zoho Writer, NTHU Checker, LanguageTool, and Ginger being the least accurate grammar checker. Despite the overall low coverage, all five grammar checkers showed very few false alarms and low rates of inaccurate correction. This suggests that the biggest pitfall for grammar checkers is the tendency to ignore mistakes rather than provide inaccurate corrections.In terms of overall performance, Grammarly outperformed the other grammar checkers on getting the highest coverage while doing great at avoiding inaccurate corrections and false alarms. Zoho Writer, which previous studies tend not to review or evaluate, also performed amazingly well. It even performed better at avoiding inaccurate corrections and false alarms than Grammarly, especially when this tool is offered entirely free on the Internet. By contrast, Ginger and LanguageTool received the worst report on both accuracy rate and the avoidance of inaccurate corrections. NTHU Checker, on the other hand, showed average performance whereas generated relatively more inaccurate corrections and false alarms than other grammar checkers.The findings also reveal the considerable variation in accuracy rate depending on error types. It is observed that grammar checkers are better at handling some error types than others. The grammatical errors that are easier to be detected and corrected include prepositions, infinitives, tense, confusion of number, and wrong position of adverbs errors. In contrast, grammar checkers show limitations on detecting un-English expressions, word order errors and errors that require more contextual information to detect and correct.Finally, the researcher provides some suggestions and methods of how teachers can incorporate grammar checkers into the classroom, including using the grammar checker for grammar instruction and writing class.