TM01. Term frequency

From tokenization, stopword filtering, stemming/lemmatization, to counting term frequency.

前言

社會科學家在做研究的過程經常會需要探究文本特徵，過去常用的方法是內容分析法，透過大量的編碼或搭配質性分析輔助軟體來分析。當我們嘗試用資訊科學的方法來解決這類的需求時，可以先想想看我們是如何「描述文本特徵」的。比方說，我們常常在日常交談中提到，某報寫的新聞經常唱衰某個政治人物、某作家的文章大量且細碎地描述周遭景物的細節、某總統候選人習慣使用「他們」來區分他者，藉以透過訊息暗示來凝聚粉絲形成我群。在這些例子中，為何我們會覺得某個媒體、某種文體或者某人的文章有一種固定的習慣、手法、操作？這是因為某些詞語的使用次數達到讓我們感到突出、明顯的地步，而這正是數量上的特徵。因此，文本探勘在做的事情就是量化文本的特徵，而最基礎的動作便是計算字詞的出現次數（詞頻），再來才是字詞間的脈絡，也就是每個字的前後關係。

文字處理流程

但在計算詞頻前，我們要先去除文本中的「雜訊」。以英文文本來說，通常會先去除代名詞（he, we, ...）、介系詞（in, to, from）、冠詞（the, a）等停用詞（Stopword），然後用stemming或lemmatization方法標準化字根，也就是去除如動名詞（-ing）、時間副詞（-ed）、比較級與最高級（-er, -est）的影響。當做完這些前處理後，便可以計算每個字的出現次數，即為詞頻，即可視為該篇文章的特徵。詞頻經常被視覺化為文字雲來使得文本特徵的概念易於被理解，也是文本分類、比較、推薦的基礎。下圖為一典型的文字處理流程，包含斷詞、篩除停用詞、詞幹萃取、計算詞頻的過程。

目標案例

本節將會用wikipedia某條目的內文來分析詞頻。輸入的文本是一篇用Python套件所取回的Wikipedia文章內容，目標輸出是按出現次數由大而小排列的詞頻，在Python裡詞頻通常會存在一個Dictionary中。

輸入：原文

Hindus (Hindustani: [ˈɦɪndu] (listen)) are persons who regard themselves as culturally, ethnically, or religiously adhering to aspects of Hinduism. Historically, the term has also been used as a geographical, cultural, and later religious identifier for people living in the Indian subcontinent.The historical meaning of the term Hindu has evolved with time. Starting with the Persian and Greek references to the land of the Indus in the 1st millennium BCE through the texts of the medieval era, the term Hindu implied a geographic, ethnic or cultural identifier for people living in the Indian subcontinent around or beyond the Sindhu (Indus) river. By the 16th century, the term began to refer to residents of the subcontinent who were not Turkic or Muslims.The historical development of Hindu self-identity within the local South Asian population, in a religious or cultural sense, is unclear. Competing theories state that Hindu identity developed in the British colonial era, or that it may have developed post-8th century CE after the Islamic invasion and medieval Hindu-Muslim wars. A sense of Hindu identity and the term Hindu appears in some texts dated between the 13th and 18th century in Sanskrit and Bengali. The 14th- and 18th-century Indian poets such as Vidyapati, Kabir and Eknath used the phrase Hindu dharma (Hinduism) and contrasted it with Turaka dharma (Islam). The Christian friar Sebastiao Manrique used the term 'Hindu' in religious context in 1649. In the 18th century, the European merchants and colonists began to refer to the followers of Indian religions collectively as Hindus, in contrast to Mohamedans for Mughals and Arabs following Islam. By the mid-19th century, colonial orientalist texts further distinguished Hindus from Buddhists, Sikhs and Jains, but the colonial laws continued to consider all of them to be within the scope of the term Hindu until about mid-20th century.

目標：詞頻

Hindu	8
term	7
century	6
Indian	4
Hindus	3
use	3
cultural	3
religious	3
text	3
colonial	3
Hinduism	2
identifier	2
people	2
live	2
historical	2
Indus	2
medieval	2
era	2
subcontinent	2
begin	2
refer	2
within	2
sense	2
identity	2
develop	2
dharma	2
contrast	2
Islam	2
Hindustani	1

0. 載入資料

載入Wikipedia的頁面內容

wikipedia為python上的一個套件，你會需要先安裝他，可以用pip install wikipedia來安裝。至於要如何使用它，可查閱https://pypi.org/project/wikipedia。

import wikipedia 
import string
cv = wikipedia.page("Taipei")
text = cv.content
print(cv.url)
print("The length of Taipei is ", len(text))

text  = wikipedia.summary("Hindus", sentences = 10)
type(text)
print("The length of Hindus summary is ", len(text))

讀取個人電腦端的靜態檔案

3KB

hindu.txt

hindu.txt 當無法用wikipedia讀取頁面時可讀取這個檔案

with open("data/hindu.txt") as fin:
    text = fin.read()

1. 斷詞 Tokenization

使用字串類別的split()函式斷詞。英文斷詞最簡單的方法是使用字串類別的split()函式，在該函式中加入據以作為斷詞依據的字元（如英文的空白），就可以把該字串切為一個由多個字詞所組成的List如下。

text_a = "What’s in a name? That which we call a rose by any other name would smell as sweet."
print(text_a.split(" "))

['What’s', 'in', 'a', 'name?', 'That', 'which', 'we', 'call', 'a', 'rose', 'by', 'any', 'other', 'name', 'would', 'smell', 'as', 'sweet.']

1.1 斷詞

將文本視為一個很長的字元List text，然後一一讀取裡面每個字元（for ch in text: ），邊讀邊存，當讀取到空白的時候（if ch == " ":），便把前面存起來的字元當成一個字串append到List tokens中。

Tokenization

print(text[:36])
tokens = []
tok = ""
for ch in text[:36]:
    if ch == " ":
        if tok:
            tokens.append(tok)
            tok = ""
            print("Tokens: ", tokens)
    else:
        tok += ch
        print(tok)

Result

Hindus (Hindustani: [ˈɦɪndu] (listen
H
Hi
Hin
Hind
Hindu
Hindus
Tokens:  ['Hindus']
(
(H
(Hi
(Hin
(Hind
(Hindu
(Hindus
(Hindust
(Hindusta
(Hindustan
(Hindustani
(Hindustani:
Tokens:  ['Hindus', '(Hindustani:']
[
[ˈ
[ˈɦ
[ˈɦɪ
[ˈɦɪn
[ˈɦɪnd
[ˈɦɪndu
[ˈɦɪndu]
Tokens:  ['Hindus', '(Hindustani:', '[ˈɦɪndu]']
(
(l
(li
(lis
(list
(liste
(listen

這邊比較難理解的應該是if tok:，我們可以寫個簡單的程式測試如下，當空字串作為if的Condition時，會相當於False，所以程式會印出"No"。就上面程式的意思是說，如果我取到一個空白字元" "，那就偵測看看，底下所取到的tok是空的還是有字，有字的話便把他加入到tokens這個List中。那沒有這個if tok:的判斷式會有什麼樣的結果？你可以改改程式看會有什麼結果。

tok = "" 
if tok: 
    print("Yes") 
else: 
    print("No")

1.2 三種斷詞方法的比較

除了使用split()法一）外，我們也可嘗試使用NLTK套件的斷詞器（方法二），或者自己寫一個相當於split()功能的斷詞函式（法三）

法一：內建函式split()

第一種方法是使用字串的內建函式split()來斷詞，該函式的輸入為據以斷詞的字元。由於英文的斷詞基準是看到空白就斷開（會有一些問題），因此，最簡單的就是把空白字元當成斷詞字元如.split(" ")。

print(text.split(" "))

法二：NLTK

第二種方法是使用NLTK的斷詞套件，NLTK是Python上的自然語言處理套件。如同下面的程式碼，第一次使用NTLK的Tokenizer應該會需要下載punkt，其為nltk套件中的Punkt tokenizer。

Punkt Sentence Tokenizer. This tokenizer divides a text into a list of sentences, by using an unsupervised algorithm to build a model for abbreviation words, collocations, and words that start sentences. It must be trained on a large collection of plaintext in the target language before it can be used.）

# import nltk
# nltk.download('punkt')
from nltk.tokenize import word_tokenize
print(word_tokenize(text))

法三：自行斷詞

第三種方法就是把上述自行斷詞的方法寫成函式

# A problematic implementation for word tokenization.
def tokenize(text):
  tokens = []
  tok = ""
  for ch in text:
    if ch == " ":
      if tok:
        tokens.append(tok)
        tok = ""
    else:
      tok += ch
  return tokens
print(tokenize(text))

這三個斷詞方法產生的結果多少會有所不同，你可以比較看看有什麼不同（練習TM01_2）。第一種當然是沒有處理標點符號的問題，用NLTK斷詞他會把標點符號作為單一字詞斷開，只要用str.isalpha()就可一一判斷每個斷開的字詞是否是英文字詞，目前自行斷詞的結果和str.split()結果是一模一樣的。但NTLK的Tokenizer和其他兩個相較下，計算速度應該會較慢，你可以上網查詢如何計算程式的執行時間。

如何計算程式的執行時間？可上網google "running time python"，便可找到在stackoverflow上的說明「How do I get time of a Python program's execution?」如下。

import time start_time = time.time() main() print("--- %s seconds ---" % (time.time() - start_time))

法一：split()結果

['Hindus', '(Hindustani:', '[ˈɦɪndu]', '(listen))', 'are', 'persons', 'who', 'regard', 'themselves', 'as', 'culturally,', 'ethnically,', 'or', 'religiously', 'adhering', 'to', 'aspects', 'of', 'Hinduism.', 'Historically,', 'the', 'term', 'has', 'also', 'been', 'used', 'as', 'a', 'geographical,', 'cultural,', 'and', 'later', 'religious', 'identifier', 'for', 'people', 'living', 'in', 'the', 'Indian', 'subcontinent.The', 'historical', 'meaning', 'of', 'the', 'term', 'Hindu', 'has', 'evolved', 'with', 'time.', 'Starting', 'with', 'the', 'Persian', 'and', 'Greek', 'references', 'to', 'the', 'land', 'of', 'the', 'Indus', 'in', 'the', '1st', 'millennium', 'BCE', 'through', 'the', 'texts', 'of', 'the', 'medieval', 'era,', 'the', 'term', 'Hindu', 'implied', 'a', 'geographic,', 'ethnic', 'or', 'cultural', 'identifier', 'for', 'people', 'living', 'in', 'the', 'Indian', 'subcontinent', 'around', 'or', 'beyond', 'the', 'Sindhu', '(Indus)', 'river.', 'By', 'the', '16th', 'century,', 'the', 'term', 'began', 'to', 'refer', 'to', 'residents', 'of', 'the', 'subcontinent', 'who', 'were', 'not', 'Turkic', 'or', 'Muslims.The', 'historical', 'development', 'of', 'Hindu', 'self-identity', 'within', 'the', 'local', 'South', 'Asian', 'population,', 'in', 'a', 'religious', 'or', 'cultural', 'sense,', 'is', 'unclear.', 'Competing', 'theories', 'state', 'that', 'Hindu', 'identity', 'developed', 'in', 'the', 'British', 'colonial', 'era,', 'or', 'that', 'it', 'may', 'have', 'developed', 'post-8th', 'century', 'CE', 'after', 'the', 'Islamic', 'invasion', 'and', 'medieval', 'Hindu-Muslim', 'wars.', 'A', 'sense', 'of', 'Hindu', 'identity', 'and', 'the', 'term', 'Hindu', 'appears', 'in', 'some', 'texts', 'dated', 'between', 'the', '13th', 'and', '18th', 'century', 'in', 'Sanskrit', 'and', 'Bengali.', 'The', '14th-', 'and', '18th-century', 'Indian', 'poets', 'such', 'as', 'Vidyapati,', 'Kabir', 'and', 'Eknath', 'used', 'the', 'phrase', 'Hindu', 'dharma', '(Hinduism)', 'and', 'contrasted', 'it', 'with', 'Turaka', 'dharma', '(Islam).', 'The', 'Christian', 'friar', 'Sebastiao', 'Manrique', 'used', 'the', 'term', "'Hindu'", 'in', 'religious', 'context', 'in', '1649.', 'In', 'the', '18th', 'century,', 'the', 'European', 'merchants', 'and', 'colonists', 'began', 'to', 'refer', 'to', 'the', 'followers', 'of', 'Indian', 'religions', 'collectively', 'as', 'Hindus,', 'in', 'contrast', 'to', 'Mohamedans', 'for', 'Mughals', 'and', 'Arabs', 'following', 'Islam.', 'By', 'the', 'mid-19th', 'century,', 'colonial', 'orientalist', 'texts', 'further', 'distinguished', 'Hindus', 'from', 'Buddhists,', 'Sikhs', 'and', 'Jains,', 'but', 'the', 'colonial', 'laws', 'continued', 'to', 'consider', 'all', 'of', 'them', 'to', 'be', 'within', 'the', 'scope', 'of', 'the', 'term', 'Hindu', 'until', 'about', 'mid-20th', 'century.']

法二：NLTK結果

['Hindus', '(', 'Hindustani', ':', '[', 'ˈɦɪndu', ']', '(', 'listen', ')', ')', 'are', 'persons', 'who', 'regard', 'themselves', 'as', 'culturally', ',', 'ethnically', ',', 'or', 'religiously', 'adhering', 'to', 'aspects', 'of', 'Hinduism', '.', 'Historically', ',', 'the', 'term', 'has', 'also', 'been', 'used', 'as', 'a', 'geographical', ',', 'cultural', ',', 'and', 'later', 'religious', 'identifier', 'for', 'people', 'living', 'in', 'the', 'Indian', 'subcontinent.The', 'historical', 'meaning', 'of', 'the', 'term', 'Hindu', 'has', 'evolved', 'with', 'time', '.', 'Starting', 'with', 'the', 'Persian', 'and', 'Greek', 'references', 'to', 'the', 'land', 'of', 'the', 'Indus', 'in', 'the', '1st', 'millennium', 'BCE', 'through', 'the', 'texts', 'of', 'the', 'medieval', 'era', ',', 'the', 'term', 'Hindu', 'implied', 'a', 'geographic', ',', 'ethnic', 'or', 'cultural', 'identifier', 'for', 'people', 'living', 'in', 'the', 'Indian', 'subcontinent', 'around', 'or', 'beyond', 'the', 'Sindhu', '(', 'Indus', ')', 'river', '.', 'By', 'the', '16th', 'century', ',', 'the', 'term', 'began', 'to', 'refer', 'to', 'residents', 'of', 'the', 'subcontinent', 'who', 'were', 'not', 'Turkic', 'or', 'Muslims.The', 'historical', 'development', 'of', 'Hindu', 'self-identity', 'within', 'the', 'local', 'South', 'Asian', 'population', ',', 'in', 'a', 'religious', 'or', 'cultural', 'sense', ',', 'is', 'unclear', '.', 'Competing', 'theories', 'state', 'that', 'Hindu', 'identity', 'developed', 'in', 'the', 'British', 'colonial', 'era', ',', 'or', 'that', 'it', 'may', 'have', 'developed', 'post-8th', 'century', 'CE', 'after', 'the', 'Islamic', 'invasion', 'and', 'medieval', 'Hindu-Muslim', 'wars', '.', 'A', 'sense', 'of', 'Hindu', 'identity', 'and', 'the', 'term', 'Hindu', 'appears', 'in', 'some', 'texts', 'dated', 'between', 'the', '13th', 'and', '18th', 'century', 'in', 'Sanskrit', 'and', 'Bengali', '.', 'The', '14th-', 'and', '18th-century', 'Indian', 'poets', 'such', 'as', 'Vidyapati', ',', 'Kabir', 'and', 'Eknath', 'used', 'the', 'phrase', 'Hindu', 'dharma', '(', 'Hinduism', ')', 'and', 'contrasted', 'it', 'with', 'Turaka', 'dharma', '(', 'Islam', ')', '.', 'The', 'Christian', 'friar', 'Sebastiao', 'Manrique', 'used', 'the', 'term', "'Hindu", "'", 'in', 'religious', 'context', 'in', '1649', '.', 'In', 'the', '18th', 'century', ',', 'the', 'European', 'merchants', 'and', 'colonists', 'began', 'to', 'refer', 'to', 'the', 'followers', 'of', 'Indian', 'religions', 'collectively', 'as', 'Hindus', ',', 'in', 'contrast', 'to', 'Mohamedans', 'for', 'Mughals', 'and', 'Arabs', 'following', 'Islam', '.', 'By', 'the', 'mid-19th', 'century', ',', 'colonial', 'orientalist', 'texts', 'further', 'distinguished', 'Hindus', 'from', 'Buddhists', ',', 'Sikhs', 'and', 'Jains', ',', 'but', 'the', 'colonial', 'laws', 'continued', 'to', 'consider', 'all', 'of', 'them', 'to', 'be', 'within', 'the', 'scope', 'of', 'the', 'term', 'Hindu', 'until', 'about', 'mid-20th', 'century', '.']

方法三：自行斷詞結果

['Hindus', '(Hindustani:', '[ˈɦɪndu]', '(listen))', 'are', 'persons', 'who', 'regard', 'themselves', 'as', 'culturally,', 'ethnically,', 'or', 'religiously', 'adhering', 'to', 'aspects', 'of', 'Hinduism.', 'Historically,', 'the', 'term', 'has', 'also', 'been', 'used', 'as', 'a', 'geographical,', 'cultural,', 'and', 'later', 'religious', 'identifier', 'for', 'people', 'living', 'in', 'the', 'Indian', 'subcontinent.The', 'historical', 'meaning', 'of', 'the', 'term', 'Hindu', 'has', 'evolved', 'with', 'time.', 'Starting', 'with', 'the', 'Persian', 'and', 'Greek', 'references', 'to', 'the', 'land', 'of', 'the', 'Indus', 'in', 'the', '1st', 'millennium', 'BCE', 'through', 'the', 'texts', 'of', 'the', 'medieval', 'era,', 'the', 'term', 'Hindu', 'implied', 'a', 'geographic,', 'ethnic', 'or', 'cultural', 'identifier', 'for', 'people', 'living', 'in', 'the', 'Indian', 'subcontinent', 'around', 'or', 'beyond', 'the', 'Sindhu', '(Indus)', 'river.', 'By', 'the', '16th', 'century,', 'the', 'term', 'began', 'to', 'refer', 'to', 'residents', 'of', 'the', 'subcontinent', 'who', 'were', 'not', 'Turkic', 'or', 'Muslims.The', 'historical', 'development', 'of', 'Hindu', 'self-identity', 'within', 'the', 'local', 'South', 'Asian', 'population,', 'in', 'a', 'religious', 'or', 'cultural', 'sense,', 'is', 'unclear.', 'Competing', 'theories', 'state', 'that', 'Hindu', 'identity', 'developed', 'in', 'the', 'British', 'colonial', 'era,', 'or', 'that', 'it', 'may', 'have', 'developed', 'post-8th', 'century', 'CE', 'after', 'the', 'Islamic', 'invasion', 'and', 'medieval', 'Hindu-Muslim', 'wars.', 'A', 'sense', 'of', 'Hindu', 'identity', 'and', 'the', 'term', 'Hindu', 'appears', 'in', 'some', 'texts', 'dated', 'between', 'the', '13th', 'and', '18th', 'century', 'in', 'Sanskrit', 'and', 'Bengali.', 'The', '14th-', 'and', '18th-century', 'Indian', 'poets', 'such', 'as', 'Vidyapati,', 'Kabir', 'and', 'Eknath', 'used', 'the', 'phrase', 'Hindu', 'dharma', '(Hinduism)', 'and', 'contrasted', 'it', 'with', 'Turaka', 'dharma', '(Islam).', 'The', 'Christian', 'friar', 'Sebastiao', 'Manrique', 'used', 'the', 'term', "'Hindu'", 'in', 'religious', 'context', 'in', '1649.', 'In', 'the', '18th', 'century,', 'the', 'European', 'merchants', 'and', 'colonists', 'began', 'to', 'refer', 'to', 'the', 'followers', 'of', 'Indian', 'religions', 'collectively', 'as', 'Hindus,', 'in', 'contrast', 'to', 'Mohamedans', 'for', 'Mughals', 'and', 'Arabs', 'following', 'Islam.', 'By', 'the', 'mid-19th', 'century,', 'colonial', 'orientalist', 'texts', 'further', 'distinguished', 'Hindus', 'from', 'Buddhists,', 'Sikhs', 'and', 'Jains,', 'but', 'the', 'colonial', 'laws', 'continued', 'to', 'consider', 'all', 'of', 'them', 'to', 'be', 'within', 'the', 'scope', 'of', 'the', 'term', 'Hindu', 'until', 'about', 'mid-20th']

1.3 只做斷詞的不足之處

['What’s', 'in', 'a', 'name?', 'That', 'which', 'we', 'call', 'a', 'rose', 'by', 'any', 'other', 'name', 'would', 'smell', 'as', 'sweet.']

類似上述斷詞結果顯然有一些令人不滿意之處，尤其是用split()或自行斷詞。這些情況包含name?後面有個問號，sweet.後面有句號，That的首字為大寫，這些狀況均會影響之後計算詞頻的結果。因此，我們得做一點處理，比方說，要去除標點符號，或者像是冠詞、代名詞、介系詞等可能在分析上無意義的詞、字首大寫的情形。在計算詞頻後的數節將會一一處理這些情形。

<練習>斷詞

1. 你可比較看看用上述自己寫的斷詞結果和用split()的結果有何不同（Hint: 計算一下兩個tokens長度是否相同）。

2. 用split()和自己撰寫tokenizer的結果都是tokens list。請問，要如何用程式來比較看看，這兩個tokens list有何不同？

2. 計算詞頻 Counting term frequency

由於NLTK可以幫忙解決標點符號的問題，所以這邊使用NLTK的tokenizer來做英文斷詞，斷詞完後使用Counter來計算詞頻，計算完後的結果用Counter的函式.most_common()來列出前20大詞頻。前20大詞頻如程式的輸出所述，以定冠詞、不定冠詞、連接詞、標點符號、介系詞等居多。從這個結果，至少可以觀察到三點問題，其一是標點符號應該要去除，其二是，冠詞、連接詞、介系詞等可能都要去除、the和The由於首字大寫的關係被視為不同的字，可能需要合併。

計算詞頻

from collections import Counter

tokens = word_tokenize(text)
word_count = Counter(tokens)

for w, c in word_count.most_common(10):
    print("%s\t%d" % (w, c))

前20大詞頻

the	35
,	33
[	30
]	30
.	18
and	16
of	14
to	12
in	12
Hindu	11
or	6
term	6
century	6
Hindus	6
a	5
The	5
as	4
for	4
Indian	4
3	4

3. 去除標點符號 Removal of Punctuation Marks

從Tokenization一節就可以發現，用字串內建函式split()以及手動斷詞的結果差不多，但和nltk做取詞的結果差非常多，最主要的差異出現在標點符號上。標點符號若利用內建函式來斷詞會被視為字詞的一部分，因而會影響詞頻統計的結果，比方說，在句末的字和該字不在句末會差一個句號而被分開計算出現次數，括號、逗號都有相同的情形。因此，調整斷詞品質的第一件事是先把標點符號給移除了。string類別本身就自帶有標點符號，可以把它印出來看看。

import string
print(string.punctuation)  
# !"#$%&'()*+,-./:;<=>?@[\]^_`{|}~

以下的程式都會用函式來撰寫。對於函式要如何設計、如何寫，可以參閱Function一節並做練習。

6. Function (60%)

以下的tokens是NLTK斷詞完後的結果，NTLK會將標點符號斷為單詞，和字母區隔開來，只需要偵測該單詞（法一）是否是標點符號，或者偵測該單詞（法二）是否是字母就好了。

法一：過濾標點符號

由於NLTK會把標點符號斷為單一tok，所以我可以用for-loop跑過所有的tokens（NLTK斷詞後的結果），逐一檢視他是否在標點符號（string.punctuation）中。要檢測某個字是否為更長的字的子字串，只要用if str_a in str_b的語法就可以偵測。以這個例子來說，他是不希望該tok為標點符號，所以要用not in。

def remove_punctuation_marks(tokens):
    clean_tokens = []
    for tok in tokens:
        if tok not in string.punctuation:
            clean_tokens.append(tok)
    return clean_tokens

print(remove_punctuation_marks(tokens))

結果如下，可以看得出來數字和'–'的符號都保留下來了。

['Rembrandt', 'Harmenszoon', 'van', 'Rijn', 'also', 'US', 'Dutch', 'ˈrɛmbrɑnt', 'ˈɦɑrmə', 'n', 'soːn', 'vɑn', 'ˈrɛin', 'listen', '15', 'July', '1606', '–', '4', 'October', '1669', 'was', 'a', 'Dutch', 'draughtsman', 'painter', 'and', 'printmaker', 'An', 'innovative', 'and', 'prolific', 'master', 'in', 'three', 'media', 'he', 'is', 'generally', 'considered', 'one', 'of', 'the', 'greatest', 'visual', 'artists', 'in', 'the', 'history', 'of', 'art', 'and', 'the', 'most', 'important', 'in', 'Dutch', 'art', 'history', 'Unlike', 'most', 'Dutch', 'masters', 'of', 'the', '17th', 'century', 'Rembrandt', "'s", 'works', 'depict', 'a', 'wide', 'range', 'of', 'style', 'and', 'subject', 'matter', 'from', 'portraits', 'and', 'self-portraits', 'to', 'landscapes', 'genre', 'scenes', 'allegorical', 'and', 'historical', 'scenes', 'and', 'biblical', 'and', 'mythological', 'themes', 'as', 'well', 'as', 'animal', 'studies', 'His', 'contributions', 'to', 'art', 'came', 'in', 'a', 'period', 'of', 'great', 'wealth', 'and', 'cultural', 'achievement', 'that', 'historians', 'call', 'the', 'Dutch', 'Golden', 'Age', 'when', 'Dutch', 'art', 'especially', 'Dutch', 'painting', 'although', 'in', 'many', 'ways', 'antithetical', 'to', 'the', 'Baroque', 'style', 'that', 'dominated', 'Europe', 'was', 'extremely', 'prolific', 'and', 'innovative', 'and', 'gave', 'rise', 'to', 'important', 'new', 'genres', 'Like', 'many', 'artists', 'of', 'the', 'Dutch', 'Golden', 'Age', 'such', 'as', 'Jan', 'Vermeer', 'of', 'Delft', 'Rembrandt', 'was', 'also', 'an', 'avid', 'art', 'collector', 'and', 'dealer', 'Rembrandt', 'never', 'went', 'abroad', 'but', 'he', 'was', 'considerably', 'influenced', 'by', 'the', 'work', 'of', 'the', 'Italian', 'masters', 'and', 'Netherlandish', 'artists', 'who', 'had', 'studied', 'in', 'Italy', 'like', 'Pieter', 'Lastman', 'the', 'Utrecht', 'Caravaggists', 'and', 'Flemish', 'Baroque', 'Peter', 'Paul', 'Rubens', 'After', 'he', 'achieved', 'youthful', 'success', 'as', 'a', 'portrait', 'painter', 'Rembrandt', "'s", 'later', 'years', 'were', 'marked', 'by', 'personal', 'tragedy', 'and', 'financial', 'hardships', 'Yet', 'his', 'etchings', 'and', 'paintings', 'were', 'popular', 'throughout', 'his', 'lifetime', 'his', 'reputation', 'as', 'an', 'artist', 'remained', 'high', 'and', 'for', 'twenty', 'years', 'he', 'taught', 'many', 'important', 'Dutch', 'painters.Rembrandt', "'s", 'portraits', 'of', 'his', 'contemporaries', 'self-portraits', 'and', 'illustrations', 'of', 'scenes', 'from', 'the', 'Bible', 'are', 'regarded', 'as', 'his', 'greatest', 'creative', 'triumphs', 'His', 'self-portraits', 'form', 'a', 'unique', 'and', 'intimate', 'biography', 'in', 'which', 'the', 'artist', 'surveyed', 'himself', 'without', 'vanity', 'and', 'with', 'the', 'utmost', 'sincerity', 'Rembrandt', "'s", 'foremost', 'contribution', 'in', 'the', 'history', 'of', 'printmaking', 'was', 'his', 'transformation', 'of', 'the', 'etching', 'process', 'from', 'a', 'relatively', 'new', 'reproductive', 'technique', 'into', 'a', 'true', 'art', 'form', 'along', 'with', 'Jacques', 'Callot']

法二：isalpha() 只把字母留下來

一樣是接續用NLTK取完tokens的結果，然後用isalpha()來偵測，看看他是否每個tok都是英文字元，如果是的話，才把這個tok加入clean_tokens中傳回來。

def remove_punctuation_marks(tokens):
    clean_tokens = []
    for tok in tokens:
        if tok.isalpha():
            clean_tokens.append(tok)
    return clean_tokens

print(remove_punctuation_marks(tokens))

法三：Pythonic Style: Comprehensive

把方法二寫成Comprehensive的寫法。

def remove_punctuation_marks(tokens):
    return [tok for tok in tokens if tok.isalpha()]

print(remove_punctuation_marks(tokens))

['Rembrandt', 'Harmenszoon', 'van', 'Rijn', 'also', 'US', 'Dutch', 'ˈrɛmbrɑnt', 'ˈɦɑrmə', 'n', 'soːn', 'vɑn', 'ˈrɛin', 'listen', 'July', 'October', 'was', 'a', 'Dutch', 'draughtsman', 'painter', 'and', 'printmaker', 'An', 'innovative', 'and', 'prolific', 'master', 'in', 'three', 'media', 'he', 'is', 'generally', 'considered', 'one', 'of', 'the', 'greatest', 'visual', 'artists', 'in', 'the', 'history', 'of', 'art', 'and', 'the', 'most', 'important', 'in', 'Dutch', 'art', 'history', 'Unlike', 'most', 'Dutch', 'masters', 'of', 'the', 'century', 'Rembrandt', 'works', 'depict', 'a', 'wide', 'range', 'of', 'style', 'and', 'subject', 'matter', 'from', 'portraits', 'and', 'to', 'landscapes', 'genre', 'scenes', 'allegorical', 'and', 'historical', 'scenes', 'and', 'biblical', 'and', 'mythological', 'themes', 'as', 'well', 'as', 'animal', 'studies', 'His', 'contributions', 'to', 'art', 'came', 'in', 'a', 'period', 'of', 'great', 'wealth', 'and', 'cultural', 'achievement', 'that', 'historians', 'call', 'the', 'Dutch', 'Golden', 'Age', 'when', 'Dutch', 'art', 'especially', 'Dutch', 'painting', 'although', 'in', 'many', 'ways', 'antithetical', 'to', 'the', 'Baroque', 'style', 'that', 'dominated', 'Europe', 'was', 'extremely', 'prolific', 'and', 'innovative', 'and', 'gave', 'rise', 'to', 'important', 'new', 'genres', 'Like', 'many', 'artists', 'of', 'the', 'Dutch', 'Golden', 'Age', 'such', 'as', 'Jan', 'Vermeer', 'of', 'Delft', 'Rembrandt', 'was', 'also', 'an', 'avid', 'art', 'collector', 'and', 'dealer', 'Rembrandt', 'never', 'went', 'abroad', 'but', 'he', 'was', 'considerably', 'influenced', 'by', 'the', 'work', 'of', 'the', 'Italian', 'masters', 'and', 'Netherlandish', 'artists', 'who', 'had', 'studied', 'in', 'Italy', 'like', 'Pieter', 'Lastman', 'the', 'Utrecht', 'Caravaggists', 'and', 'Flemish', 'Baroque', 'Peter', 'Paul', 'Rubens', 'After', 'he', 'achieved', 'youthful', 'success', 'as', 'a', 'portrait', 'painter', 'Rembrandt', 'later', 'years', 'were', 'marked', 'by', 'personal', 'tragedy', 'and', 'financial', 'hardships', 'Yet', 'his', 'etchings', 'and', 'paintings', 'were', 'popular', 'throughout', 'his', 'lifetime', 'his', 'reputation', 'as', 'an', 'artist', 'remained', 'high', 'and', 'for', 'twenty', 'years', 'he', 'taught', 'many', 'important', 'Dutch', 'portraits', 'of', 'his', 'contemporaries', 'and', 'illustrations', 'of', 'scenes', 'from', 'the', 'Bible', 'are', 'regarded', 'as', 'his', 'greatest', 'creative', 'triumphs', 'His', 'form', 'a', 'unique', 'and', 'intimate', 'biography', 'in', 'which', 'the', 'artist', 'surveyed', 'himself', 'without', 'vanity', 'and', 'with', 'the', 'utmost', 'sincerity', 'Rembrandt', 'foremost', 'contribution', 'in', 'the', 'history', 'of', 'printmaking', 'was', 'his', 'transformation', 'of', 'the', 'etching', 'process', 'from', 'a', 'relatively', 'new', 'reproductive', 'technique', 'into', 'a', 'true', 'art', 'form', 'along', 'with', 'Jacques', 'Callot']

設計好去除標點符號的函式後，便在計算詞頻前加入去除標點符號的函式看看結果，再和之前沒有去除標點符號的前10大詞頻比較看看，可以發現不受標點符號影響了。

去除標點符號並計算詞頻

tokens = remove_punctuation_marks(tokens)
word_count = Counter(tokens)

for w, c in word_count.most_common(10):
    print("%s\t%d" % (w, c))

去除標點符號後的前10詞頻

the	29
and	12
of	10
in	10
to	9
Hindu	8
term	7
or	6
century	6
as	4

原先沒去除標點符號的前10詞頻

the	29
,	17
and	12
of	10
.	10
in	10
to	9
Hindu	8
term	7
or	6

4. 篩除停用詞 Stopword filtering

停用詞的定義可參見Wikiepdia:停用詞。停用詞的偵測都是有個停用詞表，例如英文停用詞表、中文停用詞表，然後偵測斷詞後的tokens有沒有包含停用詞在裡頭。但必須注意的是，並不是所有的分析都要去除停用詞的，例如你想要知道某人的演講風格、寫作風格做歸類，那可能就要考慮納入停用詞來進行分析。

在信息檢索中，為節省存儲空間和提高搜索效率，在自然語言處理數據（或文本）之前或之後會自動過濾掉某些字或詞，這些字或詞即被稱為Stop Words(停用詞)。不要把停用詞與安全口令混淆。 這些停用詞都是人工輸入、非自動化生成的，生成後的停用詞會形成一個停用詞表。但是，並沒有一個明確的停用詞表能夠適用於所有的工具。甚至有一些工具是明確地避免使用停用詞來支持短語搜索的。

對於一個給定的目的，任何一類的詞語都可以被選作停用詞。通常意義上，停用詞大致分為兩類。一類是人類語言中包含的功能詞，這些功能詞極其普遍，與其他詞相比，功能詞沒有什麼實際含義，比如'the'、'is'、'at'、'which'、'on'等。但是對於搜尋引擎來說，當所要搜索的短語包含功能詞，特別是像The Who、The The或Take That等複合名詞時，停用詞的使用就會導致問題。另一類詞包括詞彙詞，比如'want'等，這些詞應用十分廣泛，但是對這樣的詞搜尋引擎無法保證能夠給出真正相關的搜索結果，難以幫助縮小搜索範圍，同時還會降低搜索的效率，所以通常會把這些詞從問題中移去，從而提高搜索性能。

載入英文停用詞 Loading Enlgish stopwords

先自NLTK套件載入停用詞，以下所示即為NLTK所定義的英文停用詞，注意到這些全為小寫，並且我們先把英文停用詞先存到一個stopword_list來使用。

載入英文停用詞

import nltk
from nltk.corpus import stopwords
# nltk.download('stopwords')

stopword_list = stopwords.words('english')
print(stopword_list)

英文停用詞表

注意英文停用詞表全為小寫

['i', 'me', 'my', 'myself', 'we', 'our', 'ours', 'ourselves', 'you', "you're", "you've", "you'll", "you'd", 'your', 'yours', 'yourself', 'yourselves', 'he', 'him', 'his', 'himself', 'she', "she's", 'her', 'hers', 'herself', 'it', "it's", 'its', 'itself', 'they', 'them', 'their', 'theirs', 'themselves', 'what', 'which', 'who', 'whom', 'this', 'that', "that'll", 'these', 'those', 'am', 'is', 'are', 'was', 'were', 'be', 'been', 'being', 'have', 'has', 'had', 'having', 'do', 'does', 'did', 'doing', 'a', 'an', 'the', 'and', 'but', 'if', 'or', 'because', 'as', 'until', 'while', 'of', 'at', 'by', 'for', 'with', 'about', 'against', 'between', 'into', 'through', 'during', 'before', 'after', 'above', 'below', 'to', 'from', 'up', 'down', 'in', 'out', 'on', 'off', 'over', 'under', 'again', 'further', 'then', 'once', 'here', 'there', 'when', 'where', 'why', 'how', 'all', 'any', 'both', 'each', 'few', 'more', 'most', 'other', 'some', 'such', 'no', 'nor', 'not', 'only', 'own', 'same', 'so', 'than', 'too', 'very', 's', 't', 'can', 'will', 'just', 'don', "don't", 'should', "should've", 'now', 'd', 'll', 'm', 'o', 're', 've', 'y', 'ain', 'aren', "aren't", 'couldn', "couldn't", 'didn', "didn't", 'doesn', "doesn't", 'hadn', "hadn't", 'hasn', "hasn't", 'haven', "haven't", 'isn', "isn't", 'ma', 'mightn', "mightn't", 'mustn', "mustn't", 'needn', "needn't", 'shan', "shan't", 'shouldn', "shouldn't", 'wasn', "wasn't", 'weren', "weren't", 'won', "won't", 'wouldn', "wouldn't"]

移除停用詞 Remove stopwords from the tokens.

在斷詞再去除標點符號後，會得到一個新的tokens list，接下來就是用for-loop一一偵測tokens中的每個tok是否在stopword_list中，如果tok不在停用詞中的話，就把該tok append到tokens_clean中。但觀看結果顯然會受大小寫影響，尤其是句首的定冠詞和不定冠詞，以及I等主詞，因此，在移除前要先做英文的大小寫轉換。

判斷是否為停用詞

def remove_stopwords(tokens):
    tokens_clean = []
    for tok in tokens:
        if tok not in stopword_list:
            tokens_clean.append(tok)
    return tokens_clean

print(remove_stopwords(tokens))

去除停用詞的結果

注意到以下若是大寫的停用詞並沒有被移除，如句首的定冠詞和不定冠詞。

['Hindus', 'Hindustani', 'ˈɦɪndu', 'listen', 'persons', 'regard', 'culturally', 'ethnically', 'religiously', 'adhering', 'aspects', 'Hinduism', 'Historically', 'term', 'also', 'used', 'geographical', 'cultural', 'later', 'religious', 'identifier', 'people', 'living', 'Indian', 'historical', 'meaning', 'term', 'Hindu', 'evolved', 'time', 'Starting', 'Persian', 'Greek', 'references', 'land', 'Indus', 'millennium', 'BCE', 'texts', 'medieval', 'era', 'term', 'Hindu', 'implied', 'geographic', 'ethnic', 'cultural', 'identifier', 'people', 'living', 'Indian', 'subcontinent', 'around', 'beyond', 'Sindhu', 'Indus', 'river', 'By', 'century', 'term', 'began', 'refer', 'residents', 'subcontinent', 'Turkic', 'historical', 'development', 'Hindu', 'within', 'local', 'South', 'Asian', 'population', 'religious', 'cultural', 'sense', 'unclear', 'Competing', 'theories', 'state', 'Hindu', 'identity', 'developed', 'British', 'colonial', 'era', 'may', 'developed', 'century', 'CE', 'Islamic', 'invasion', 'medieval', 'wars', 'A', 'sense', 'Hindu', 'identity', 'term', 'Hindu', 'appears', 'texts', 'dated', 'century', 'Sanskrit', 'Bengali', 'The', 'Indian', 'poets', 'Vidyapati', 'Kabir', 'Eknath', 'used', 'phrase', 'Hindu', 'dharma', 'Hinduism', 'contrasted', 'Turaka', 'dharma', 'Islam', 'The', 'Christian', 'friar', 'Sebastiao', 'Manrique', 'used', 'term', 'religious', 'context', 'In', 'century', 'European', 'merchants', 'colonists', 'began', 'refer', 'followers', 'Indian', 'religions', 'collectively', 'Hindus', 'contrast', 'Mohamedans', 'Mughals', 'Arabs', 'following', 'Islam', 'By', 'century', 'colonial', 'orientalist', 'texts', 'distinguished', 'Hindus', 'Buddhists', 'Sikhs', 'Jains', 'colonial', 'laws', 'continued', 'consider', 'within', 'scope', 'term', 'Hindu', 'century']

5. 去除大小寫影響 Handle Capitalization in English

處理英文的大小寫轉換有兩種作法，Solution 1是設計一個lowercase()函式把所有tokens都先轉成小寫，然後再移去停用詞，這樣做會導致所有文字都被轉為小寫，這樣後續計算詞頻不管是否詞彙在句首都會列計到同一個詞；Solution 2 是在偵測是否為停用詞時臨時轉小寫，這樣的結果是不會把原本的內容全部轉為小寫，但句首大寫的詞在算詞頻時會被列計為另外一個詞。

Solution 1. 全部先轉為小寫

全部轉小寫後加入新的List

def lowercase(tokens):
    tokens_lower = []
    for tok in tokens:
        tokens_lower.append(tok.lower())
    return tokens_lower

print(remove_stopwords(lowercase(tokens)))

output

['hindus', 'hindustani', 'ˈɦɪndu', 'listen', 'persons', 'regard', 'culturally', 'ethnically', 'religiously', 'adhering', 'aspects', 'hinduism', 'historically', 'term', 'also', 'used', 'geographical', 'cultural', 'later', 'religious', 'identifier', 'people', 'living', 'indian', 'historical', 'meaning', 'term', 'hindu', 'evolved', 'time', 'starting', 'persian', 'greek', 'references', 'land', 'indus', 'millennium', 'bce', 'texts', 'medieval', 'era', 'term', 'hindu', 'implied', 'geographic', 'ethnic', 'cultural', 'identifier', 'people', 'living', 'indian', 'subcontinent', 'around', 'beyond', 'sindhu', 'indus', 'river', 'century', 'term', 'began', 'refer', 'residents', 'subcontinent', 'turkic', 'historical', 'development', 'hindu', 'within', 'local', 'south', 'asian', 'population', 'religious', 'cultural', 'sense', 'unclear', 'competing', 'theories', 'state', 'hindu', 'identity', 'developed', 'british', 'colonial', 'era', 'may', 'developed', 'century', 'ce', 'islamic', 'invasion', 'medieval', 'wars', 'sense', 'hindu', 'identity', 'term', 'hindu', 'appears', 'texts', 'dated', 'century', 'sanskrit', 'bengali', 'indian', 'poets', 'vidyapati', 'kabir', 'eknath', 'used', 'phrase', 'hindu', 'dharma', 'hinduism', 'contrasted', 'turaka', 'dharma', 'islam', 'christian', 'friar', 'sebastiao', 'manrique', 'used', 'term', 'religious', 'context', 'century', 'european', 'merchants', 'colonists', 'began', 'refer', 'followers', 'indian', 'religions', 'collectively', 'hindus', 'contrast', 'mohamedans', 'mughals', 'arabs', 'following', 'islam', 'century', 'colonial', 'orientalist', 'texts', 'distinguished', 'hindus', 'buddhists', 'sikhs', 'jains', 'colonial', 'laws', 'continued', 'consider', 'within', 'scope', 'term', 'hindu', 'century']

Solution 2. 在比較判斷時才轉小寫

在判斷停用詞時才轉小寫

def remove_stopwords(tokens):
    tokens_clean = []
    for tok in tokens:
        if tok.lower() not in stopword_list:
            tokens_clean.append(tok)
    return tokens_clean
print(remove_stopwords(tokens)

output

['Hindus', 'Hindustani', 'ˈɦɪndu', 'listen', 'persons', 'regard', 'culturally', 'ethnically', 'religiously', 'adhering', 'aspects', 'Hinduism', 'Historically', 'term', 'also', 'used', 'geographical', 'cultural', 'later', 'religious', 'identifier', 'people', 'living', 'Indian', 'historical', 'meaning', 'term', 'Hindu', 'evolved', 'time', 'Starting', 'Persian', 'Greek', 'references', 'land', 'Indus', 'millennium', 'BCE', 'texts', 'medieval', 'era', 'term', 'Hindu', 'implied', 'geographic', 'ethnic', 'cultural', 'identifier', 'people', 'living', 'Indian', 'subcontinent', 'around', 'beyond', 'Sindhu', 'Indus', 'river', 'century', 'term', 'began', 'refer', 'residents', 'subcontinent', 'Turkic', 'historical', 'development', 'Hindu', 'within', 'local', 'South', 'Asian', 'population', 'religious', 'cultural', 'sense', 'unclear', 'Competing', 'theories', 'state', 'Hindu', 'identity', 'developed', 'British', 'colonial', 'era', 'may', 'developed', 'century', 'CE', 'Islamic', 'invasion', 'medieval', 'wars', 'sense', 'Hindu', 'identity', 'term', 'Hindu', 'appears', 'texts', 'dated', 'century', 'Sanskrit', 'Bengali', 'Indian', 'poets', 'Vidyapati', 'Kabir', 'Eknath', 'used', 'phrase', 'Hindu', 'dharma', 'Hinduism', 'contrasted', 'Turaka', 'dharma', 'Islam', 'Christian', 'friar', 'Sebastiao', 'Manrique', 'used', 'term', 'religious', 'context', 'century', 'European', 'merchants', 'colonists', 'began', 'refer', 'followers', 'Indian', 'religions', 'collectively', 'Hindus', 'contrast', 'Mohamedans', 'Mughals', 'Arabs', 'following', 'Islam', 'century', 'colonial', 'orientalist', 'texts', 'distinguished', 'Hindus', 'Buddhists', 'Sikhs', 'Jains', 'colonial', 'laws', 'continued', 'consider', 'within', 'scope', 'term', 'Hindu', 'century']

6. 完整計算詞頻（大小寫影響、停用詞、標點符號）

前面我們一共做了三個程序來去除大小寫、停用詞、標點符號的影響，而在大小寫轉換上有兩種Solution，可以比較看看這兩種Solution所跑出來的詞頻結果。在這個資料案例裡，只有專有名詞大小寫有差別，詞頻數量看似並沒有差別。你覺得哪一種轉法比較好呢？我比較常用第二種轉法。

Sol.1 先全轉小寫

注意到我是用lowercase()這個函式把tokens整個List轉小寫。

word_count = Counter(remove_stopwords(lowercase(tokens)))

for w, c in word_count.most_common(20):
    print("%s\t%d" % (w, c))

Sol.1 詞頻

hindu	8
term	7
century	6
indian	4
hindus	3
used	3
cultural	3
religious	3
texts	3
colonial	3
hinduism	2
identifier	2
people	2
living	2
historical	2
indus	2
medieval	2
era	2
subcontinent	2
began	2

Sol.2 比較時暫時轉

word_count = Counter(remove_stopwords(tokens))

for w, c in word_count.most_common(20):
    print("%s\t%d" % (w, c))

Sol.2 詞頻

Hindu	8
term	7
century	6
Indian	4
Hindus	3
used	3
cultural	3
religious	3
texts	3
colonial	3
Hinduism	2
identifier	2
people	2
living	2
historical	2
Indus	2
medieval	2
era	2
subcontinent	2
began	2

7. 詞幹提取 Stemming

詞幹提取要解決的問題：接下來的Stemming和Lemmatization都是為了去除字尾動名詞、過去式/完成式、和比較級／最高級的影響，例如若只是單純的詞頻統計，沒有要看文句結構的話，那start、 started、starting應該要被算為同一個字。可是前面的做法會把這三個字當成不同的字來計算詞頻，因此要想辦法解決這樣的問題。

何謂Stemming？以下說明內容來自Wikipedia:詞幹提取。但這部分Wikiepdia:Stemming英文的解說會比中文解說清楚。從下面wikipedia所舉的例子可以看出來，詞幹提取的結果會處理掉現在分詞、過去分詞、比較級字尾，進而使得這些詞可以被當同一個詞來統計。

In linguistic morphology and information retrieval, stemming is the process of reducing inflected (or sometimes derived) words to their word stem, base or root form—generally a written word form. The stem need not be identical to the morphological root of the word; it is usually sufficient that related words map to the same stem, even if this stem is not in itself a valid root. Algorithms for stemming have been studied in computer science since the 1960s. Many search engines treat words with the same stem as synonyms as a kind of query expansion, a process called conflation.

Example. A stemmer for English operating on the stem cat should identify such strings as cats, catlike, and catty. A stemming algorithm might also reduce the words fishing, fished, and fisher to the stem fish. The stem need not be a word, for example the Porter algorithm reduces, argue, argued, argues, arguing, and argus to the stem argu.

底下詞幹提取的方法使用了NLTK套件的Snowball演算法。Stemming with Snowball algorithm implemented by NLTK. Reference: http://snowball.tartarus.org/texts/introduction.html。詞幹提取的時機是在移除停用詞後。但底下目前這個版本的移除停用詞並無解決大小寫的問題，你可以自行解決（輸出中只列出了詞頻2以上的結果）。

Stemming Code

from nltk.stem.snowball import SnowballStemmer
snowball_stemmer = SnowballStemmer("english")

stemmed_tokens = []
for tok in remove_stopwords(tokens):
    stemmed_tokens.append(snowball_stemmer.stem(tok))
word_count = Counter(stemmed_tokens)

for w, c in word_count.most_common(50):
    print("%s\t%d" % (w, c))

term freq.

dutch	9
rembrandt	6
art	6
artist	5
master	3
histori	3
import	3
portrait	3
scene	3
mani	3
also	2
painter	2
printmak	2
innov	2
prolif	2
greatest	2
work	2
style	2
genr	2
studi	2
contribut	2
achiev	2
golden	2
age	2
paint	2
baroqu	2
new	2
like	2
year	2
etch	2
form	2

8. 詞型還原 Lemmatization

類似Stemming的另一個方法式Lemmatization。詞型還原的演算法可以解決動詞、名詞、形容詞或副詞等詞，並會依照大小寫來進行判斷。Stemming比較單純是把詞尾、詞綴給去除，但Lemmatization則是讀取WordNet詞網的資料，透過字庫比對來抽取詞彙的原型。所以Lemmatization在不是-ed型的過去分詞尤其會表現得更好。

詞形還原測試

我們先來寫些簡單的Code測試一下Lemmatization與Stemming的效果。

Lemmatize簡例

# import nltk
# nltk.download('wordnet')
from nltk.stem import WordNetLemmatizer
wordnet_lemmatizer = WordNetLemmatizer()

def lemmatize(token):
    # ADJ (a), ADJ_SAT (s), ADV (r), NOUN (n) or VERB (v)
    for p in ['v', 'n', 'a', 'r', 's']:
        l = wordnet_lemmatizer.lemmatize(token, pos=p)
        if l != token:
            return l
    return token

print(lemmatize('Dogs')) # Dogs
print(lemmatize('dogs')) # dog
print(lemmatize('hits')) # hit

比較stemming和lemmatization

for w in [
    'open', 'opens', 'opened', 'opening', 'unopened',
    'talk', 'talks', 'talked', 'talking',
    'decompose', 'decomposes', 'decomposed', 'decomposing',
    'do', 'does', 'did', 
    'wrote', 'written', 'ran', 'gave', 'held', 'went', 'gone',
    'lied', 'lies', 'lay', 'lain', 'lying', 
    'cats', 'people', 'feet', 'women', 'smoothly', 'firstly', 'secondly', 
    'install', 'installed', 'uninstall',
    'internalization', 'internationalization',
    'decontextualization', 'decontextualized', 'decentralization', 'decentralized']:
    s = snowball_stemmer.stem(w)
    l = lemmatize(w)
    if s != l:
        print("%s\t%s\t%s" % (w, s, l))

測試結果

Original Stemming Lemmatization
unopened	unopen	unopened
decompose	decompos	decompose
decomposes	decompos	decompose
decomposed	decompos	decompose
decomposing	decompos	decompose
does	doe	do
did	did	do
wrote	wrote	write
written	written	write
ran	ran	run
gave	gave	give
held	held	hold
went	went	go
gone	gone	go
lain	lain	lie
people	peopl	people
feet	feet	foot
women	women	woman
smoothly	smooth	smoothly
firstly	first	firstly
secondly	second	secondly
install	instal	install
uninstall	uninstal	uninstall
internalization	intern	internalization
internationalization	internation	internationalization
decontextualization	decontextu	decontextualization
decontextualized	decontextu	decontextualized
decentralization	decentr	decentralization
decentralized	decentr	decentralize

詞形還原後計算詞頻

和Stemming一樣，是把Lemmatization做在移除停用詞後，然後就進入詞頻的計算。

Lemmatization並計算詞頻

lemmatized_tokens = []
for tok in remove_stopwords(tokens):
    lemmatized_tokens.append(lemmatize(tok))
word_count = Counter(lemmatized_tokens)

for w, c in word_count.most_common(50):
    print("%s\t%d" % (w, c))

詞頻結果

Dutch	9
Rembrandt	6
art	6
artist	5
master	3
great	3
history	3
important	3
portrait	3
scene	3
many	3
also	2
painter	2
innovative	2
prolific	2
work	2
style	2
genre	2
study	2
contribution	2
Golden	2
Age	2
paint	2
Baroque	2
new	2
year	2
etch	2
form	2

9. Application: WordCloud

以上文字處理過程，最常見的應用就是文字雲。文字雲通常把字詞依照其出現次數重複繪製後，丟入線上文字雲的軟體就可以自動產生依照詞頻的文字雲，如https://www.jasondavies.com/wordcloud/。

產生繪製雲字雲的文字

repeated_tokens = []
for w, c in word_count.most_common():
    for i in range(c):
        repeated_tokens.append(w)
print(" ".join(repeated_tokens))

output

hindu hindu hindu hindu hindu hindu hindu hindu term term term term term term term century century century century century century indian indian indian indian hindus hindus hindus used used used cultural cultural cultural religious religious religious texts texts texts colonial colonial colonial hinduism hinduism identifier identifier people people living living historical historical indus indus medieval medieval era era subcontinent subcontinent began began refer refer within within sense sense identity identity developed developed dharma dharma islam islam hindustani ˈɦɪndu listen persons regard culturally ethnically religiously adhering aspects historically also geographical later meaning evolved time starting persian greek references land millennium bce implied geographic ethnic around beyond sindhu river residents turkic development local south asian population unclear competing theories state british may ce islamic invasion wars appears dated sanskrit bengali poets vidyapati kabir eknath phrase contrasted turaka christian friar sebastiao manrique context european merchants colonists followers religions collectively contrast mohamedans mughals arabs following orientalist distinguished buddhists sikhs jains laws continued consider scope