Natural Language Processing in Python: 5 Useful Libraries!

1. NLTK – The ‘Big Brother’ of Natural Language Processing

NLTK (Natural Language Toolkit) is arguably the most classic natural language processing library. It is like a toolbox filled with various tools for processing text.

Run the following Python code:

import nltk
from nltk.tokenize import word_tokenize
from nltk.corpus import stopwords

# Download necessary data
nltk.download('punkt')
nltk.download('stopwords')

# Example text
text = "Python is a very elegant programming language!"

# Tokenization
tokens = word_tokenize(text)
print("Tokenization result:", tokens)

# Stop word filtering
stop_words = set(stopwords.words('english'))
filtered_words = [word for word in tokens if word.lower() not in stop_words]
print("After filtering stop words:", filtered_words)

1. SpaCy – Speed and Elegance Combined

SpaCy is a more modern natural language processing library, especially suitable for handling large-scale texts.

Run the following Python code:

import spacy

# Load Chinese model
nlp = spacy.load('zh_core_web_sm')

# Analyze text
text = "Xiaoming studies artificial intelligence at Peking University"
doc = nlp(text)

# Output part-of-speech tagging
for token in doc:
    print(f"{token.text}: {token.pos_}")

# Named entity recognition
for ent in doc.ents:
    print(f"Entity: {ent.text}, Type: {ent.label_}")

1. Jieba – The Expert in Chinese Word Segmentation

For us Chinese users, Jieba word segmentation is simply a magic tool!

Run the following Python code:

import jieba

# Basic word segmentation
text = "I love programming in Python"
words = jieba.cut(text)
print("Basic segmentation:", "/".join(words))

# Add custom dictionary
jieba.add_word("Python programming")
words = jieba.cut(text)
print("After adding dictionary:", "/".join(words))

1. Transformers – Embrace the AI Era

The Transformers library allows us to easily use various advanced pre-trained models.

Run the following Python code:

from transformers import pipeline

# Sentiment analysis
classifier = pipeline("sentiment-analysis")
result = classifier("I love programming in Python!")[0]
print(f"Sentiment analysis result: {result['label']}, Confidence: {result['score']:.2f}")

# Text generation
generator = pipeline("text-generation")
text = generator("Python is", max_length=30, num_return_sequences=1)
print("Generated text:", text[0]['generated_text'])

1. TextBlob – A Simple and User-Friendly Text Processing Tool

TextBlob is a particularly friendly library, especially suitable for beginners.

Run the following Python code:

from textblob import TextBlob

# Create TextBlob object
text = "Python is awesome! I love coding."
blob = TextBlob(text)

# Sentiment analysis
print("Sentiment polarity:", blob.sentiment.polarity)
print("Subjectivity:", blob.sentiment.subjectivity)

# Part-of-speech tagging
print("Part-of-speech tagging:", blob.tags)

Tips:

1. Remember to install these libraries using pip before using them!
2. When processing Chinese, it is recommended to use Jieba or SpaCy’s Chinese model.
3. Large models may require better hardware support; it is advisable to start practicing with small datasets.

Notes:

• First-time users of NLTK need to download the relevant data packages.
• SpaCy’s models need to be downloaded separately.
• Be mindful of memory usage when using Transformers.

Friends, today’s Python learning journey ends here! Each of these 5 libraries has its own characteristics, and I suggest you choose one that interests you to study in depth. Remember to get hands-on coding, and feel free to ask me any questions in the comments section. I wish you all happy learning, and may your Python skills improve day by day!