Original link: https://lisz.me/tech/webmaster/modules-python.html
foreword
In recent years, thanks to its lightweight, easy-to-learn and easy-to-use, third-party support and many libraries, the Python language has been widely used in research and project development related to machine learning. In academia, there are all-round machine learning libraries represented by Scikit-Learn ; in the industry, there are production-level machine learning model computing frameworks represented by TensorFlow and PyTorch . (Of course, computing frameworks such as PyTorch are also used in academia to actually build large-scale deep learning models.) But for most people, when learning these libraries and frameworks or using them to engage in certain research and project development, it is possible Still using my laptop or desktop. Even in college laboratories, this kind of thing is not uncommon. Therefore, Anaconda, which has an interactive interface and is relatively easy to use, may be your first choice for managing the Python environment.
When we are writing some code in Python, and it is impossible to get results in just a few seconds or a few minutes at a time, it is particularly effective to submit tasks to high-performance workstations or cluster operating systems. Especially when the application scale is large and the number of calculation iterations is large, the non-interactive job submission method will become more beneficial. After all, if you use your own notebook to run such a large calculation, the resources are basically taken up by the calculation, and there is no way to use the notebook to do other things. It is even said that computing will also increase the temperature of core components such as the CPU, thereby affecting computing performance. In this comparison, I have to say how wise it is to submit tasks to high-performance workstations or cluster operating systems.
In fact, Anaconda can still be used on server operating systems without an interactive interface. We can use its free lite version – miniconda . Although miniconda is already a streamlined version, it still has a lot more than the native Python environment. From the perspective of the high-performance computing environment, using Modules to directly manage the Python environment is actually closer to the original, and it is more conducive to users to use it with other environments. For example, to install Python’s MPI support library – MPI4PY, you only need to load the two basic environments of Python and MPI through the Modules management tool, and use pip3 install mpi4py command to install.
practice
The steps to bring the Python environment into Modules management are two steps: the first step is to compile the source code and install it; the second step is to add the Modules configuration file. Of course, at the beginning, you still need to confirm whether the compilation environment is complete and whether the folder is ready.
Environment and folder preparation
# 安装编译环境sudo apt install -y build-essential libbz2-dev libdb-dev \ libreadline-dev libffi-dev libgdbm-dev liblzma-dev \ libncursesw5-dev libsqlite3-dev libssl-dev \ zlib1g-dev uuid-dev tk-dev wget # 准备文件夹sudo mkdir -p /opt/python/3.10.6
Compile and install
# 下载源代码cd /tmp wget -c https://www.python.org/ftp/python/3.10.6/Python-3.10.6.tar.xz # 解压源代码tar xf Python-3.10.6.tar.xz # 配置安装路径及编译选项cd Python-3.10.6 ./configure --prefix = /opt/python/3.10.6 --enable-optimizations --with-lto # 编译及安装make && sudo make install
Configure Modules
sudo mkdir -p /opt/modules/modulefiles/py sudo vim /opt/modules/modulefiles/py/3.10.6
First prepare the folder as above command, and create a new module configuration file, the content is as follows:
#%Module proc ModulesHelp { } { puts stderr \t This module file will load Python 3.10.6 " } module-whatis " Enable Python 3.10.6 " eval set [ array get env HOME ] set basedir /opt/python/3.10.6 prepend-path PATH " ${ basedir } /bin " prepend-path LIBRARY_PATH " ${ basedir } /lib " prepend-path LD_LIBRARY_PATH " ${ basedir } /lib " prepend-path INCLUDE_PATH " ${ basedir } /include " prepend-path LD_INCLUDE_PATH " ${ basedir } /include "
verify
# 查看所有可用模块╰─ $ module ava -------------------------- /opt/modules/modulefiles --------------------------- dot module-git module-info modules null py/3.10.6 use.own Key: modulepath
# 加载python 3.10.6 环境，并确认已加载模块╰─ $ module load py/3.10.6 ╰─ $ module list Currently Loaded Modulefiles: 1 ) py/3.10.6
# 确认目前python 版本╰─ $ python3 -V Python 3.10.6
use
Since the above operation installed Python 3.10.6 in a system folder, the following warning prompt will appear after the compilation is completed. But don’t worry, ordinary users can use it normally through the venv virtual environment tool.
Installing collected packages: setuptools, pip WARNING: The scripts pip3 and pip3.10 are installed in '/opt/python/3.10.6/bin' which is not on PATH. Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location . Successfully installed pip-22.2.1 setuptools-63.2.0 WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
As shown below, after the python 3.10.6 module has been loaded as in the verification section, use the following command to create a virtual environment, activate it and use it.
When creating a new virtual environment, the last parameter env refers to the name of the virtual environment. We can take any string that meets the python rules as the virtual environment name. It is worth noting that the files related to the python virtual environment will be installed in the folder with the same name under the current directory where the command is executed. For ease of management and use, it is recommended to place all python virtual environments in the same directory.
# 新建env 虚拟环境╰─ $ python3 -m venv env # 激活env 虚拟环境╰─ $ source env /bin/activate # 可以看到<env> 的环境提示# 尝试升级pip，可以看到成功升级╭─zhonger@lep-u ~ ‹env› ╰─ $ pip3 install -U pip Requirement already satisfied: pip in ./env/lib/python3.10/site-packages ( 22.2.1 ) Collecting pip Using cached pip-23.0.1-py3-none-any.whl ( 2.1 MB ) Installing collected packages: pip Attempting uninstall: pip Found existing installation: pip 22.2.1 Uninstalling pip-22.2.1: Successfully uninstalled pip-22.2.1 Successfully installed pip-23.0.1 # 取消激活env 虚拟环境╰─ $ deactivate
interesting question
If the virtual environment is created using the python module provided by module, do I still need to use module to load the python module when actually running the virtual environment? The answer is no . The essence of a virtual environment is to copy the necessary files needed to run the same command. Compare the top-level directories of the python module and the env virtual environment as follows. It can be found that there is not much difference between the two. The env virtual environment lacks the share directory and has more pyvenv.cfg files. Looking at this file, there are declarations of relationships to python modules. Check the bin directory again, you can see that the python executable command is linked, and the pip command is directly copied from the original python module. This then allows ordinary users to manage pip commands and python libraries themselves.
╰─ $ ls /opt/python/3.10.6 bin include lib share ╰─ $ ls env bin include lib lib64 pyvenv.cfg ╰─ $ cat env /pyvenv.cfg home = /opt/python/3.10.6/bin include-system-site-packages = false version = 3.10.6 ╰─ $ ll env /bin total 36K -rw-r--r-- 1 zhonger zhonger 8.9K Mar 20 15:20 Activate.ps1 -rw-r--r-- 1 zhonger zhonger 2.0K Mar 20 15:20 activate -rw-r--r-- 1 zhonger zhonger 908 Mar 20 15:20 activate.csh -rw-r--r-- 1 zhonger zhonger 2.1K Mar 20 15:20 activate.fish -rwxrwxr-x 1 zhonger zhonger 234 Mar 20 15:20 pip -rwxrwxr-x 1 zhonger zhonger 234 Mar 20 15:20 pip3 -rwxrwxr-x 1 zhonger zhonger 234 Mar 20 15:20 pip3.10 lrwxrwxrwx 1 zhonger zhonger 7 Mar 20 15:20 python -> python3 lrwxrwxrwx 1 zhonger zhonger 30 Mar 20 15:20 python3 -> /opt/python/3.10.6/bin/python3 lrwxrwxrwx 1 zhonger zhonger 7 Mar 20 15:20 python3.10 -> python3
References
- Official Python Documentation – Building Python
- Official Python Documentation – Performance Options
- Official Python Documentation – Virtual Environments and Packages
- Python in Ubuntu environment
This article is transferred from: https://lisz.me/tech/webmaster/modules-python.html
This site is only for collection, and the copyright belongs to the original author.