更新内容（2018-12-1）：
一些论文的复现代码已开源在GIthub上：
地址：https://github.com/shillyshallysxy/Learning_NLP
前段时间在看Doc2Vec，但一段时间的浏览后发现大多博客是对原论文的思路进行讲解，对于其具体如何的实现以及更细节一些的东西，博主还是懵懵懂懂，想着大概是大家都心领神会了。为了给和博主有一样情况的朋友一些帮助，翻阅了一些材料，将Doc2Vec中的PV-DM的一种实现方法写了下来，以便大家直观的看到这个模型是怎么工作的（一种工作方式）以便大家理解论文中的思路。
Doc2Vec论文的链接：https://cs.stanford.edu/~quocle/paragraph_vector.pdf
想要阅读原论文的读者，可以点击上面的链接。
看到这里的读者，博主姑且默认已经知道Word2Vec的工作方式了，按照博主的理解，Word2Vec的工作简单阐述就是将单词转换为词向量，并且这些词向量可以一定程度的表达单词之间的语义关系，各种方法各有优劣，具体在这里不再赘述。
如果还不知道Word2Vec是什么，关于Word2Vec的知识，这位博主的博客给了我很多的帮助，链接：https://blog.csdn.net/itplus/article/details/37969519，相信研读完能有很大的帮助。
或者参考该论文：https://arxiv.org/abs/1310.4546
论文思路是在word2vec的基础上，加入段落主题这个属性的考量
盗用原论文中的图片，即：
举一个例子，当在段落1中 描述的是 “大家出门郊游野餐”，段落2描述的是 ”我在开会“这类主题 时，在给出上下文 “我坐在__” 让模型预测下一个单词。
对于Word2Vec来说，就不会考量段落1或是段落2的语境和主题，模型也许会认为 “坐在椅子上” 在训练集中出现的最多，所以大概是坐在 “椅子上” 最有可能？他也不知道。 但对于Doc2Vec来说，同时跟随上下文输入的还有一个代表了本段落的主题的Vector，比如当一起输入的是代表着段落1的Vector时，那很有可能预测是 “草上" 或 ”地上“ 这类单词，当然也有可能是坐在 ”椅子上“， 当一起输入的是段落2的Vector时，更有可能预测是坐在 ”椅子上“ ，开会的时候 ”坐在草上“ 或是 ”坐在地上“ 就显得不太合理（笑）。
按照段落区分主题，这里给出了一个比较简单的方法：
声明几个变量 embedding_size（词向量的长度, 可以根据自己的需求自行设置，越大单词就被映射到越高维的空间，表达能力越强），vocabulary_size（总单词种类的数量），每个单词都有自己的词向量 shape = [1, embedding_size]
如上图，Word2Vec中的average方法，输入上下文词向量，得到输入向量 shape = [1, embedding_size]（average方法，所以求平均），经过权重矩阵 shape = [embedding_size, vocabulary_size] 和偏置矩阵 shape = [1, vocabulary_size] 通过Nec-loss得到预测输出每个单词的概率的向量 shape = [1, vocabulary_size] 。
至于Doc2Vec，这里只需要对Word2Vec进行一些小更改就行了，按照不同的段落，新增一个矩阵，段落向量矩阵 shape = [doc_size, doc_embedding_size]，doc_embedding_size（段落向量的长度, 可以根据自己的需求自行设置），doc_size（文中段落的数量）。
每个段落拥有自己的段落向量 shape = [1, doc_embedding_size]，对于每次训练，这里选择联结(concatenate)的方式，将Word2Vec中的输入词向量与输入词向量所属的段落向量联结，最终输入向量变为 shape = [1, embedding_size+doc_embedding_size]，同样，小小的更改权重矩阵，经过权重矩阵 shape = [embedding_size+doc_embedding_size, vocabulary_size] 和偏置矩阵 shape = [1, vocabulary_size] 通过Nec-loss得到预测输出每个单词的概率的向量 shape = [1, vocabulary_size]。
训练过程就是不断更新 段落向量矩阵（shape = [doc_size, doc_embedding_size]） 和 词向量矩阵（[Vocabulary_size, embedding_size]） 的过程了。训练出来的 embedding 就可以用于后续的 比如情感预测，垃圾短信过滤之类的任务中了！
这样就完成了一个简单的Doc2Vec !
希望对在学习中的大家有所帮助。
声明一些变量和方法：
sess = tf.Session() # Declare model parameters batch_size = 500 vocabulary_size = 7500 generations = 100000 model_learning_rate = 0.001 embedding_size = 200 doc_embedding_size = 100 concatenated_size = embedding_size + doc_embedding_size num_sampled = int(batch_size/2) # Number of negative examples to sample. window_size = 3 def normalize_text(texts, stops): texts = [x.lower() for x in texts] texts = [''.join(c for c in x if c not in string.punctuation) for x in texts] texts = [''.join(c for c in x if c not in '0123456789') for x in texts] texts = [' '.join([word for word in x.split() if word not in (stops)]) for x in texts] texts = [' '.join(x.split()) for x in texts] return(texts) # Build dictionary of words def build_dictionary(sentences, vocabulary_size): split_sentences = [s.split() for s in sentences] words = [x for sublist in split_sentences for x in sublist] count = [['RARE', -1]] count.extend(collections.Counter(words).most_common(vocabulary_size-1)) word_dict = {} for word, word_count in count: word_dict[word] = len(word_dict) return(word_dict) # Turn text data into lists of integers from dictionary def text_to_numbers(sentences, word_dict): # Initialize the returned data data = [] for sentence in sentences: sentence_data = [] for word in sentence.split(): if word in word_dict: word_ix = word_dict[word] else: word_ix = 0 sentence_data.append(word_ix) data.append(sentence_data) return(data) # Generate data randomly (N words behind, target, N words ahead) def generate_batch_data(sentences, batch_size, window_size, method='skip_gram'): # Fill up data batch batch_data = [] label_data = [] while len(batch_data) < batch_size: rand_sentence_ix = int(np.random.choice(len(sentences), size=1)) rand_sentence = sentences[rand_sentence_ix] window_sequences = [rand_sentence[max((ix-window_size),0):(ix+window_size+1)] for ix, x in enumerate(rand_sentence)] label_indices = [ix if ix<window_size else window_size for ix,x in enumerate(window_sequences)] if method=='doc2vec': batch_and_labels = [(rand_sentence[i:i+window_size], rand_sentence[i+window_size]) for i in range(0, len(rand_sentence)-window_size)] batch, labels = [list(x) for x in zip(*batch_and_labels)] batch = [x + [rand_sentence_ix] for x in batch] else: raise ValueError('Method {} not implemented yet.'.format(method)) batch_data.extend(batch[:batch_size]) label_data.extend(labels[:batch_size]) batch_data = batch_data[:batch_size] label_data = label_data[:batch_size] batch_data = np.array(batch_data) label_data = np.transpose(np.array([label_data])) return(batch_data, label_data) # Load the movie review data # Check if data was downloaded, otherwise download it and save for future use def load_movie_data(): save_folder_name = 'temp' pos_file = os.path.join(save_folder_name, 'rt-polaritydata', 'rt-polarity.pos') neg_file = os.path.join(save_folder_name, 'rt-polaritydata', 'rt-polarity.neg') # Check if files are already downloaded if not os.path.exists(os.path.join(save_folder_name, 'rt-polaritydata')): movie_data_url = 'http://www.cs.cornell.edu/people/pabo/movie-review-data/rt-polaritydata.tar.gz' # Save tar.gz file req = requests.get(movie_data_url, stream=True) with open('temp_movie_review_temp.tar.gz', 'wb') as f: for chunk in req.iter_content(chunk_size=1024): if chunk: f.write(chunk) f.flush() # Extract tar.gz file into temp folder tar = tarfile.open('temp_movie_review_temp.tar.gz', "r:gz") tar.extractall(path='temp') tar.close() pos_data = [] with open(pos_file, 'r', encoding='latin-1') as f: for line in f: pos_data.append(line.encode('ascii',errors='ignore').decode()) f.close() pos_data = [x.rstrip() for x in pos_data] neg_data = [] with open(neg_file, 'r', encoding='latin-1') as f: for line in f: neg_data.append(line.encode('ascii',errors='ignore').decode()) f.close() neg_data = [x.rstrip() for x in neg_data] texts = pos_data + neg_data target = [1]*len(pos_data) + [0]*len(neg_data) return(texts, target)主体：
# load data texts, target = text_helpers.load_movie_data() # Normalize text texts = text_helpers.normalize_text(texts, stops) # Texts must contain at least 3 words target = [target[ix] for ix, x in enumerate(texts) if len(x.split()) > window_size] texts = [x for x in texts if len(x.split()) > window_size] # Build our data set and dictionaries word_dictionary = text_helpers.build_dictionary(texts, vocabulary_size) word_dictionary_rev = dict(zip(word_dictionary.values(), word_dictionary.keys())) text_data = text_helpers.text_to_numbers(texts, word_dictionary) # Define Embeddings: embeddings = tf.Variable(tf.random_uniform([vocabulary_size, embedding_size], -1.0, 1.0)) doc_embeddings = tf.Variable(tf.random_uniform([len(texts), doc_embedding_size], -1.0, 1.0)) # NCE loss parameters nce_weights = tf.Variable(tf.truncated_normal([vocabulary_size, concatenated_size], stddev=1.0 / np.sqrt(concatenated_size))) nce_biases = tf.Variable(tf.zeros([vocabulary_size])) # Create placeholders x_inputs = tf.placeholder(tf.int32, shape=[None, window_size + 1]) # plus 1 for doc index y_target = tf.placeholder(tf.int32, shape=[None, 1]) valid_dataset = tf.constant(valid_examples, dtype=tf.int32) # Lookup the word embedding # Add together element embeddings in window: embed = tf.zeros([batch_size, embedding_size]) for element in range(window_size): embed += tf.nn.embedding_lookup(embeddings, x_inputs[:, element]) doc_indices = tf.slice(x_inputs, [0,window_size],[batch_size,1]) doc_embed = tf.nn.embedding_lookup(doc_embeddings,doc_indices) # concatenate embeddings final_embed = tf.concat(axis=1, values=[embed, tf.squeeze(doc_embed)]) # Get loss from prediction loss = tf.reduce_mean(tf.nn.nce_loss(weights=nce_weights, biases=nce_biases, labels=y_target, inputs=final_embed, num_sampled=num_sampled, num_classes=vocabulary_size)) # Create optimizer optimizer = tf.train.GradientDescentOptimizer(learning_rate=model_learning_rate) train_step = optimizer.minimize(loss) # Cosine similarity between words norm = tf.sqrt(tf.reduce_sum(tf.square(embeddings), 1, keep_dims=True)) normalized_embeddings = embeddings / norm valid_embeddings = tf.nn.embedding_lookup(normalized_embeddings, valid_dataset) similarity = tf.matmul(valid_embeddings, normalized_embeddings, transpose_b=True) # Create model saving operation saver = tf.train.Saver({"embeddings": embeddings, "doc_embeddings": doc_embeddings}) #Add variable initializer. init = tf.global_variables_initializer() sess.run(init) for i in range(generations): batch_inputs, batch_labels = text_helpers.generate_batch_data(text_data, batch_size, window_size, method='doc2vec') feed_dict = {x_inputs : batch_inputs, y_target : batch_labels} # Run the train step sess.run(train_step, feed_dict=feed_dict)
