我编写了下面的代码来尝试 gensim 的 word2vec 实现。我有两个问题:
- 尽管我删除了停用词,但“the”这个词被列为与“friend”最相似的词之一。
- 最相似的“朋友”一词并不令人满意(至少根据我的主观评价)。我应该尝试更大的文本(austen-emma.txt 文件包含 192427 个单词)还是问题出在其他地方?
谢谢。
import nltk
from nltk.tokenize import sent_tokenize
from nltk.corpus import gutenberg
import gensim
from gensim.models import Word2Vec
from gensim.parsing.preprocessing import remove_stopwords
from nltk.tokenize import RegexpTokenizer
text = gutenberg.raw('austen-emma.txt').
text = remove_stopwords(text).
tokenizer = RegexpTokenizer(r'\w+').
data = [].
for i in sent_tokenize(text):
temp = [].
for j in tokenizer.tokenize(i):
temp.append(j.lower()).
data.append(temp).
model = gensim.models.Word2Vec(data, min_count = 1,
size = 32, window = 2)
model.wv.most_similar(positive='friend', topn=10)
[('mind', 0.9998476505279541),
('present', 0.9998302459716797),
('till', 0.9998292326927185),
('herself', 0.9998183250427246),
('highbury', 0.999806821346283),
('the', 0.9998062252998352),
('place', 0.9998047351837158),
('house', 0.999799907207489),
('her', 0.9997915029525757),
('me', 0.9997879266738892)]