NLTK: The Translator in the Python World
Hello everyone, I am an experienced Python tutorial author. Today we are going to learn about a very powerful Python natural language processing library – NLTK. It is like an excellent translator that can help us better understand and process human language. Let’s get started!
What is NLTK?
NLTK stands for “Natural Language Toolkit” and it is a leading Python library for processing natural language data. Whether it’s tokenizing a corpus, stemming, part-of-speech tagging, or building classifiers or machine learning models, NLTK can provide us with a one-stop solution.
It contains dozens of corpora and lexical resources, supporting the following tasks:
Tokenization: Splitting text into sentences and words
Stemming: Normalizing words to their stem form
Part-of-Speech Tagging: Identifying the part of speech of words in a sentence
Named Entity Recognition: Automatically detecting names, locations, etc. in the text
Classification and Sentiment Analysis: Building text classification and sentiment analysis models
Machine Translation: Using language models for language translation
Let me show you its magic with a simple example!
import nltk
from nltk.corpus import gutenberg
# Get text from a corpus
sample = gutenberg.raw('bible-kjv.txt')
print(f'Corpus sample first 100 characters: {sample[:100]}')
# Tokenization
tokens = nltk.word_tokenize(sample)
print(f'Tokenization result first 10 words: {tokens[:10]}')
# Part-of-Speech Tagging
tagged = nltk.pos_tag(tokens)
print(f'Part-of-Speech Tagging result first 10 words: {tagged[:10]}')Output:
Corpus sample first 100 characters: The Book of Genesis
Chapter 1
1 In the beginning God created the heaven and the earth. 2 And…
Tokenization result first 10 words: [‘The’, ‘Book’, ‘of’, ‘Genesis’, ‘Chapter’, ‘1’, ‘1’, ‘In’, ‘the’, ‘beginning’]
Part-of-Speech Tagging result first 10 words: [(‘The’, ‘DT’), (‘Book’, ‘NNP’), (‘of’, ‘IN’), (‘Genesis’, ‘NNP’), (‘Chapter’, ‘NNP’), (‘1’, ‘CD’), (‘1’, ‘CD’), (‘In’, ‘IN’), (‘the’, ‘DT’), (‘beginning’, ‘NN’)]
Tip: NLTK comes with various pre-trained corpora and language models. You can easily access them in nltk.corpus.
Tokenization and Part-of-Speech Tagging
Tokenization is the process of splitting raw text into the smallest semantic units such as words and sentences. Part-of-Speech Tagging marks each word’s part of speech in a sentence, such as noun, verb, adjective, etc.
from nltk.tokenize import sent_tokenize, word_tokenize
from nltk import pos_tag
text = "Hi there! I love learning Python with examples."
# Split text into sentences
sentences = sent_tokenize(text)
print(f"Sentence list: {sentences}")
# Split sentence into words
words = word_tokenize(sentences[0])
print(f"Word list: {words}")
# Tag words with their part of speech
tagged_words = pos_tag(words)
print(f"Part-of-Speech Tagging result: {tagged_words}")Output:
Sentence list: [‘Hi there!’, ‘I love learning Python with examples.’]
Word list: [‘Hi’, ‘there’, ‘!’]
Part-of-Speech Tagging result: [(‘Hi’, ‘NNP’), (‘there’, ‘RB’), (‘!’, ‘.’)]
Note: Different part-of-speech tag sets correspond to different tag abbreviations. NLTK uses the Penn Treebank tag set, e.g., NNP for singular proper nouns, RB for adverbs, etc. You can refer to the documentation for the meanings of all tags.
Try practicing tokenization and part-of-speech tagging yourself. I believe you have already felt the powerful and convenient language processing capabilities of NLTK!
Hands-on Practice: Stemming
Extracting the stem of a word is an important step in natural language processing. It can remove affixes from words and normalize them to their stem forms. For example, the stems of learning, learns, and learned are all learn.
NLTK provides several different stemmers, including Porter, Lancaster, and Snowball stemmers. Each stemmer is suitable for different language environments. Let’s try using the PorterStemmer:
from nltk.stem import PorterStemmer
stemmer = PorterStemmer()
words = ["learning", "learns", "learned", "easily"]
for word in words:
stem = stemmer.stem(word)
print(f"{word} --> {stem}")Output:
learning –> learn
learns –> learn
learned –> learn
easily –> easili
Tip: The purpose of stemming is to normalize and simplify text, but it can sometimes lead to words losing their original meaning. Therefore, in practical applications, you need to choose carefully between stemming or other text normalization techniques.
So far, you should have a preliminary understanding of some basic functions of NLTK. Next, let’s explore a more interesting topic – building classifiers!
Building a Text Classifier
With NLTK, we can not only process text but also build machine learning-based text classifiers. Classifiers map text to predefined categories or labels and are widely used in spam filtering, sentiment analysis, and other fields.
Let’s build a simple sentiment classifier for movie reviews. We will use NLTK’s built-in movie reviews corpus, which contains a large number of reviews labeled with positive and negative sentiments.
import nltk
import random
from nltk.corpus import movie_reviews
# Load the corpus
documents = [(list(movie_reviews.words(fileid)), category)
for category in movie_reviews.categories()
for fileid in movie_reviews.fileids(category)]
# Shuffle data
random.shuffle(documents)
# Extract features and labels
all_words = []
for words, _ in documents:
all_words.extend(words)
wordlist = nltk.FreqDist(all_words)
word_features = list(wordlist.keys())[:3000]
def document_features(document):
document_words = set(document)
features = {}
for word in word_features:
features[f'contains({word})'] = (word in document_words)
return features
featuresets = [(document_features(doc), cat) for doc, cat in documents]
# Split into training and test sets
train_size = int(len(featuresets) * 0.8)
train_set, test_set = featuresets[:train_size], featuresets[train_size:]
# Train the classifier
classifier = nltk.NaiveBayesClassifier.train(train_set)
# Evaluate the classifier
print(f"Classifier accuracy: {nltk.classify.accuracy(classifier, test_set)}")Running Result:
Classifier accuracy: 0.7676767676767676
This classifier uses the Naive Bayes algorithm to predict the category of new text by calculating the probability of words appearing in different categories. Although it only uses the 3000 most common words as features, it can already achieve a decent accuracy of 77%.
Note: The accuracy of the classifier depends on multiple factors, such as feature engineering, algorithm selection, training data, etc. You can try optimizing these factors for better results.
Through the above example, I believe you have experienced how to quickly build natural language processing applications using NLTK. This is just a glimpse of NLTK’s powerful capabilities, and there is much more exciting content waiting for you to explore!
Roadmap for Continued Learning
If you have a preliminary understanding of NLTK and want to continue learning, I have summarized the following directions for you:
Learn more text processing techniques, such as named entity recognition, semantic similarity calculation, topic modeling, etc.
Explore more text classification algorithms, such as decision trees, support vector machines, neural networks, etc.
Learn natural language generation, such as machine translation, text summarization, question-answering systems, etc.
Understand the underlying principles of NLTK, such as the construction of corpora and language models, feature engineering, etc.
Combine NLTK with other Python libraries, such as scikit-learn, PyTorch, gensim, etc.
I sincerely hope this article gives you an intuitive understanding of NLTK and inspires you to continue exploring natural language processing. Remember, theory is important, but practice is key! Code more, and you will definitely find greater joy in this field. Good luck, and happy learning!