Source:Big Data DT (ID:bigdatadt)Authors:Li Mingjiang, Zhang Liangjun, Zhou Dongping, Zhang ShangjiaContent excerpted from “Python 3 Quick Start to Intelligent Data Analysis”This article is about5200 words, suggested reading time10 minutes.
This article summarizes commonly used Python libraries.
[ Guide ]Python, as a well-designed programming language, is now widely used in various fields. With its powerful third-party libraries, Python can play a significant role in various domains.
1. Numerical ComputingNumerical computing is the foundation of data mining and machine learning. Python offers various powerful extension libraries for numerical computing, commonly used numerical computing libraries are as follows.
1. NumPySupports multi-dimensional arrays and matrix operations, and provides a large number of mathematical functions for array operations. Typically used with SciPy and Matplotlib, it supports more types of numerical data than Python itself, with the most important object defined being the n-dimensional array type called ndarray, which describes a collection of elements of the same type, accessible via zero-based indexing.2. SciPyBuilt on the NumPy library, it adds a multitude of library functions commonly used in mathematics, science, and engineering calculations, such as linear algebra, numerical solutions of ordinary differential equations, signal processing, image processing, and sparse matrices, allowing for interpolation, signal filtering, and computation acceleration using C language.3. PandasA tool based on NumPy, born to solve data analysis tasks. It includes a large number of libraries and some standard data models, providing tools needed for efficiently operating large datasets and a wealth of functions and methods for quick and convenient data handling, offering great support for time series analysis, and providing various data structures such as Series, Time-Series, DataFrame, and Panel.2. Data VisualizationData visualization is an effective means to display and understand data, commonly used Python data visualization libraries are as follows.
4. MatplotlibThe first Python visualization library, many other libraries are built on its foundation or directly call this library, making it easy to obtain a general idea of the data, very powerful yet quite complex.5. SeabornUtilizes Matplotlib to create beautiful charts with concise code. The main difference from Matplotlib is that the default plotting style and color matching have a modern aesthetic.6. ggplotBased on R’s ggplot2 library, it also utilizes concepts from “The Grammar of Graphics,” allowing for the overlay of different layers to complete a plot, not suitable for creating highly personalized images, sacrificing complexity for ease of use.7. BokehLike ggplot, Bokeh is also based on the concept of “The Grammar of Graphics.” Unlike ggplot, it is entirely based on Python rather than referencing R. It excels in creating interactive charts that can be used directly on the web. Charts can be output as JSON objects, HTML documents, or interactive web applications.
Bokeh also supports data streams and real-time data, providing three levels of control for different users:
- The highest control level is for quick plotting, mainly used for creating common images;
- The medium control level, like Matplotlib, allows developers to control the basic elements of the image (e.g., points in a scatter plot);
- The lowest control level is mainly aimed at developers and software engineers. There are no defaults, and every element of the chart must be defined.
8. PlotlyCan be used via Python notebook, like Bokeh, focusing on the creation of interactive charts, but offers several chart types that are nearly absent in other libraries, such as contour plots, tree maps, and 3D charts.9. pygalLike Bokeh and Plotly, it provides interactive images that can be directly embedded in web browsers. The main difference from the other two is that charts can be output in SVG format, all charts are encapsulated as methods, and the default style is also quite beautiful, allowing for the easy creation of attractive charts with just a few lines of code.10. geoplotlibA toolbox for creating maps and geographic data. It can create various maps, such as contour area maps, heat maps, and point density maps. Pyglet (an object-oriented programming interface) must be installed for use.11. missingnoQuickly evaluates the situation of missing data visually, can sort or filter data based on completeness, or correct data based on heat maps or dendrograms.3. Web DevelopmentWeb application development is currently one of the most important parts of software development. Python offers various web development frameworks to help users quickly implement functional development. Commonly used Python web development libraries are as follows.
12. DjangoAn advanced Python web framework that supports rapid development, providing everything needed from template engines to ORM, when building apps with this library, one must follow Django’s conventions.13. SocketA low-level library for socket communication, used to establish TCP or UDP connections between servers and clients, sending requests and responses through the connection.14. FlaskA lightweight framework (microframework) based on Werkzeug and Jinja 2, comes with Jinja template engine by default, also includes other template engines or ORMs for choice, suitable for writing API services (RESTful services).15. TwistedAn event-driven network engine framework implemented in Python, built on deferred objects, a high-performance engine achieved through asynchronous architecture, not suitable for writing conventional web apps, more suitable for low-level networking.16. TornadoA Python web framework and asynchronous networking library developed by FriendFeed, using a non-blocking network I/O model, capable of handling thousands of network connections. For long polling, WebSockets, and other apps requiring long-term real-time connections, Tornado is an ideal web framework, situated between Django and Flask, effectively addressing the C10K problem.4. Database ManagementA database is the main tool for enterprises to store data, database management includes data definition, data operation, database operation management, data organization, database protection, database maintenance, etc. Python provides interfaces for all mainstream relational database management systems, commonly used Python MySQL connection libraries and their introductions are as follows.
17. MySQL-pythonAlso known as MySQLdb, it is the most popular driver for connecting Python to MySQL, many frameworks are also developed based on this library. It only supports Python 2.x, and there are many prerequisites for installation. Since this library is developed in C language, installation on Windows platforms is very unfriendly, often resulting in failures, and is now generally not recommended for use, with derivatives as replacements.18. mysqlclientFully compatible with MySQLdb, it also supports Python 3.x, is a dependency tool for Django ORM, allowing native SQL to operate on databases, installation method is the same as MySQLdb.19. PyMySQLA pure Python implementation of the driver, slower than MySQLdb, with the main feature being a simple installation method, and is also compatible with MySQL-python.20. SQLAlchemyA tool that supports both native SQL and ORM. ORM is a mapping relationship between Python objects and database tables, which can effectively speed up coding while being compatible with various database systems such as SQLite, MySQL, PostgreSQL, at the cost of some performance loss.5. Automation OperationsOperations mainly include ensuring the long-term stable operation of business, ensuring data security and reliability, and automating deployment tasks. Python can meet the vast majority of automation operation needs, and applications implemented in Linux operations using Python are as follows.
21. jumpserverAn open-source jump server (bastion host) system written in Python, implementing basic functions of a jump server, including authentication, authorization, and auditing, integrating Ansible, batch commands, etc. Supports WebTerminal Bootstrap, with a beautiful interface, automatically collects hardware information, supports recording playback, command search, real-time monitoring, batch upload and download, etc., managed via SSH protocol, no agent installation required on the client side. Mainly used to solve visual security management problems, fully open-source, easy to redevelop.22. Magedu Distributed Monitoring SystemAn automated monitoring system developed in Python, capable of monitoring common system services, applications, and network devices, can monitor multiple different services on one host, with different monitoring intervals for different services, and different monitoring intervals and alarm thresholds for the same service on different hosts, providing a data visualization interface.23. Magedu’s CMDBA hardware management system developed in Python, including hardware data collection, API, and page management functionalities, mainly used for the automated management of common devices like laptops and routers. The server’s client collects hardware data, sends the hardware information to the API, which is responsible for saving the acquired data into the database, the background management program is responsible for configuring and displaying server information.24. Task Scheduling SystemA task scheduling system developed in Python, mainly used to automate the distribution of a service process across multiple machines and multiple processes, with a service process acting as a scheduler relying on network communication to complete this task.25. Python Operations Process SystemA platform written in Python for scheduling and monitoring workflows, internally used to create, monitor, and adjust data pipelines. Allows workflow developers to easily create, maintain, and periodically schedule workflows, including numerous cross-department use cases such as data storage, growth analysis, email sending, A/B testing, etc.6. GUI ProgrammingGUI (Graphical User Interface) refers to a computer operating user interface displayed in graphical form. Python provides multiple libraries for GUI programming, commonly used Python GUI libraries are as follows.
26. TkinterA standard GUI library for Python, allowing for the rapid creation of GUI applications, usable on most UNIX platforms, as well as Windows and Macintosh systems. Tkinter 8.0 and later versions can achieve local window styles and run well on most platforms.27. wxPythonAn open-source cross-platform GUI library wxWidgets’ Python wrapper and module, an excellent GUI graphics library for Python, allowing programmers to easily create complete, functional GUI user interfaces.28. PyQtA library for creating GUI applications, a successful integration of the Python programming language and Qt, can run on all major operating systems, including UNIX, Windows, and Mac. PyQt uses a dual-license, allowing developers to choose between GPL and commercial licenses, since version 4, the GPL license can be used on all supported platforms.29. PySideA cross-platform application framework Qt’s Python binding version, providing similar functionality to PyQt, compatible API, but differs from PyQt in that it uses LGPL licensing.7. Machine LearningPython, as an ideal integration language, binds various technologies together, providing users with more convenient functions, and serves as an ideal glue platform, connecting developers with low-level integrators of external libraries to implement more efficient algorithms in C/C++. For researchers, using Python programming allows for quick code migration and modifications without spending too much energy on code revisions and standards. Developers have encapsulated many excellent dependency libraries in Python, among which the NumPy and SciPy libraries provide the standard configuration needed to solve machine learning problems.Python currently integrates a large number of machine learning frameworks, commonly used machine learning libraries are as follows.
30. Scikit-LearnScikit-Learn, based on NumPy and SciPy, is a Python module specifically built for machine learning, providing a wealth of tools for data mining and analysis, including data preprocessing, cross-validation, algorithms, and visualization algorithms.The basic functionalities of Sklearn can be divided into six parts:
- Classification
- Regression
- Clustering
- Dimensionality Reduction
- Model Selection
- Data Preprocessing
It integrates a large number of algorithms for classification, regression, and clustering, including support vector machines, logistic regression, naive Bayes, random forests, Gradient Boosting, K-means, and DBSCAN.31. Orange3Orange3 is a component-based data mining and machine learning software suite that supports scripting development in Python. It includes a series of data visualization, retrieval, preprocessing, and modeling techniques, featuring a good user interface, and can also be used as a Python module.Users can analyze data through data visualization, including statistical distribution charts, bar charts, scatter plots, as well as deeper decision trees, hierarchical clustering, heat maps, MDS (multidimensional scaling), linear prediction, etc., and can use various additional functional components provided by Orange for NLP, text mining, network analysis, inferring high-frequency datasets, and association rule data analysis.32. XGBoostXGBoost is a machine learning library focused on gradient boosting algorithms, gaining widespread attention for its excellent learning performance and efficient training speed. XGBoost supports parallel processing, improving performance by more than ten times compared to the Scikit-Learn library that also implements gradient boosting algorithms. XGBoost can handle various tasks such as regression, classification, and ranking.33. NuPICNuPIC is a machine learning platform focused on time series, with its core algorithm being the HTM algorithm, which is closer to the operational structure of the human brain compared to deep learning. The theoretical basis of the HTM algorithm mainly comes from the operation principles of the neocortex part of the human brain that processes higher cognitive functions. NuPIC can be used for prediction and anomaly detection, applicable to a wide range of scenarios, only requiring time series input.34. MilkMilk (Machine Learning Toolkit) is a machine learning toolkit in Python.Milk focuses on improving runtime speed and reducing memory usage, so most performance-sensitive code is written in C++, providing a Python interface on top of it for convenience. It emphasizes supervised classification methods, such as SVMs, KNN, random forests, and decision trees, and also supports unsupervised learning algorithms, such as K-means and affinity propagation.8. Deep LearningDeep learning, as a branch of machine learning, has shone brightly. Due to Python’s ease of use and extensibility, many deep learning frameworks provide Python interfaces, among which the following deep learning libraries are relatively popular.
35. CaffeCaffe (Convolutional Architecture for Fast Feature Embedding) is a deep learning framework centered on expressiveness, speed, and modularity, known for its clarity, readability, and rapid features, widely applied in video and image processing.In Caffe, the network structure and optimization are defined in configuration files, making it easy to get started without needing to build networks through code; training speed is fast, capable of training large datasets and state-of-the-art models; modular components can be easily extended to new models and learning tasks.36. TheanoTheano, born in 2008, is a high-performance symbolic computation and deep learning library, considered one of the pioneers of deep learning libraries and an important standard for deep learning research and applications. Its core is a compiler for mathematical expressions, designed for computing large-scale neural network training.Theano integrates well with NumPy, allowing direct use of NumPy’s ndarray, significantly lowering the learning cost of the API interface; it has good computational stability, capable of accurately computing output values of very small functions, such as log(1+x); it can dynamically generate C or CUDA code for compiling into efficient machine code.37. TensorFlowTensorFlow is a relatively high-level machine learning library, with its core code written in C++, supporting automatic differentiation, allowing users to easily design neural network structures without needing to write C++ or CUDA code or solving gradients through backpropagation. Due to the underlying implementation in C++, efficiency is guaranteed, simplifying the complexity of online deployment.
In addition to the core C++ interface, TensorFlow also has official Python, Go, and Java interfaces, allowing users to experiment with Python on a well-configured machine, and deploy models in resource-constrained embedded environments or low-latency environments using C++.TensorFlow is not limited to neural networks; its data flow graphs also support very flexible algorithm expressions and can easily implement machine learning algorithms beyond deep learning.38. KerasKeras is a highly modular neural network library implemented in Python, which can run simultaneously on TensorFlow and Theano.Keras specializes in deep learning, providing the most convenient API to date, allowing users to design neural networks by simply piecing together high-level modules, greatly reducing programming overhead and cognitive overhead. Keras supports both convolutional networks and recurrent networks, and allows switching from CPU to GPU acceleration without any code changes. While simplifying programming complexity, it does not compromise performance compared to TensorFlow and Theano.
About the authors:
Li Mingjiang, senior big data expert, executive director of Guizhou Computer Society, member of Qiannan Big Data Expert Committee, president of Qiannan Computer Society, expert in Qiannan Education Informatization Construction Expert Database, dean of the School of Computer and Information at Qiannan Normal University for Nationalities, director of the National University Big Data Education Alliance.
Zhang Liangjun, senior big data mining and analysis expert, pattern recognition expert, AI technology expert. With over 10 years of experience in big data mining and analysis, proficient in data mining and analysis using technologies such as Python, R, Hadoop, Matlab, and has in-depth research on data analysis driven by AI technologies such as machine learning.
This article is excerpted from “Python 3 Quick Start to Intelligent Data Analysis,” published with authorization from the publisher.
Editor: Wang Jing
Proofreader: Lin Yilin
