文章目录

1. 机械分词算法1.1. 正向最大匹配算法1.2. 逆向最大匹配算法

参考链接：

/lcwdzl/article/details/78493637

/liu_zhlai/article/details/52125174?spm=1001..3001.5501

代码源码地址：

/lankuohsing/Study_NLP

1. 机械分词算法

1.1. 正向最大匹配算法

custom_dict={"南京","南京市","市长","长江","大桥","江大桥",}input_sentence="南京市长江大桥"max_word_len=0for word in custom_dict:if len(word)>max_word_len:max_word_len=len(word)if len(input_sentence)<max_word_len:max_word_len=len(input_sentence)start=0seg_results=[]while start<len(input_sentence):temp_len=max_word_lenif len(input_sentence)-start<max_word_len:temp_len=len(input_sentence)-startwhile temp_len>0:sub_sentence=input_sentence[start:start+temp_len]if sub_sentence in custom_dict:seg_results.append(sub_sentence)start+=temp_lenbreakelse:temp_len-=1# 没有子串匹配，则单独成词if temp_len==0:seg_results.append(input_sentence[start:start+1])start+=1print(seg_results)

1.2. 逆向最大匹配算法

custom_dict={"南京","南京市","市长","长江","大桥","江大桥"}input_sentence="南京市长江大桥"max_word_len=0for word in custom_dict:if len(word)>max_word_len:max_word_len=len(word)if len(input_sentence)<max_word_len:max_word_len=len(input_sentence)end=len(input_sentence)seg_results=[]while end>0:temp_len=max_word_lenif end<max_word_len:temp_len=endwhile temp_len>0:sub_sentence=input_sentence[end-temp_len:end]if sub_sentence in custom_dict:seg_results.append(sub_sentence)end-=temp_lenbreakelse:temp_len-=1# 没有子串匹配，则单独成词if temp_len==0:sub_sentence=input_sentence[end-1:end]seg_results.append(sub_sentence)end-=1print(seg_results)