電腦生成的新聞有多真?─文字自動生成技術運用於經濟新聞的評估
How Genuine is Computer-Generated News? ─ Evaluation of Automated Text Generation Applied to Economic News
國立臺灣師範大學圖書資訊學研究所(Graduate Institute of Library and Information Studies, National Taiwan Normal University, Taipei, Taiwan)
摘要
本研究以經濟新聞為範圍,探討GPT-2模型,在經過約30萬篇新聞訓練後產生15篇電腦生成之新聞(CGN),混合15篇人類撰寫之新聞(HCN),由12位受試者進行1到5分的可信度評價。結果在15篇HCN中,有1篇其平均可信度為2.92,不及3,原因為沒有邏輯、主觀性強等;在15篇CGN中,有2篇其平均可信度皆為3.33,大於3,原因為內容合理、細節符合邏輯。此2篇的部分內容與語料庫比對後,發現電腦移花接木再加潤飾的能力,已可欺騙專業人士。本研究也訓練BERT模型,以瞭解自動偵測電腦生成新聞之可能性。結果上述30篇新聞中,BERT只有2篇CGN預測錯誤,其餘皆正確,比受試者們集體的預測,有5篇錯誤,成效還要高。較大規模的實驗,顯示BERT的成效,可達0.96的F1分數。
Abstract
This research explores the GPT-2 deep learning model for economic news generation and evaluation. After training GPT-2 by about 300,000 pieces of news with a total of 150 million words, 15 news articles are generated by GPT-2. Together with 15 real news articles written by journalists, 12 subjects were invited to judge the credibility of the 30 news articles with 1 to 5 scales. As a result, 8 subjects who graduated from economic-related major were more capable of discriminating the human-composed news (HCN) from the computer-generated news (CGN); while 4 subjects who graduated from non-economic related major had poor discriminating ability, and one was even unable to tell the HCN from the CGN. Among the 15 HCN articles, 1 was rated as non-genuine news, with an average credibility of 2.92, which is less than 3, due to lack of logic and strong subjectivity. Among the 15 CGN articles, 2 were rated as genuine news, with average credibility of 3.33, which is greater than 3, because the content is reasonable and the details are logical. After comparing these two articles with the corpus, it is found that the computer's ability to substitute and retouch can deceive professionals. However, most of the CGN articles have been spotted, mainly because of obvious flaws in facts and incorrect digits such as dates and stock codes. The research also explores the possibility of automatically detecting computer-generated news using BERT-based neural network model. As a result, BERT had only 2 false predictions out of the above 30 news articles. Compared with the collective prediction by the 12 subjects with 5 errors, BERT performs better. Further large-scale experiments show that the effectiveness of BERT can reach an F-score of 0.96.
關鍵詞
- 類型
- 論文
- 期刊
- 圖書資訊學刊
- ISSN
- 1606-7509
- 卷期
- 19卷1期
- 出版
- 2021年06月
- 頁碼
- 043–065
- 編號
- NTUEJ00004533
同期目錄
- 資料探勘方法於圖書資訊學領域之運用Marie Katsurai, Soohyung Joo・頁 001–017
- 使用者於資訊架構要素注意力分配之文化差異鄭瑋(Wei Jeng);胡馨元(Hsin-Yuan Hu);湯高銘(Gao-Ming Tang);簡士鎰(Shih-Yi Chien)・頁 019–041
- 電腦生成的新聞有多真?─文字自動生成技術運用於經濟新聞的評估曾元顯(Yuen-Hsien Tseng);林郁綺(Yu-Chi Lin)・頁 043–065
- 電影情緒療癒功能之質化研究─以生涯發展困擾之臺大學生為例陳書梅(Chen Su-may Sheih)・頁 067–113
- 臺灣社會科學獲補助國際期刊論文之特性探討董蕙茹(Huei-Ru Dong)・頁 115–137
- 以閱讀關鍵因素與動機視角探索網路小說閱讀生態許哲睿(Che-Jui Hsu);吳怡瑾(I-Chin Wu)・頁 139–181
- 今傳諸本系統的《秘書省續編到四庫闕書目》版本源流考楊勝祥(Sheng-Xiang Yang)・頁 183–218