Close file descriptors opened by the child process

Original link: http://ponder.work/2022/08/30/close-subprocess-opened-fd/

When we test the code, because we need to restart the service frequently, we often find that the service port is occupied.
Generally, killing the background process is ok, but if the service starts some resident background programs, the port may not be released.

In UNIX-like systems, all open files and ports are abstracted as file descriptors
Since python3.4, file descriptors are non-inheritable by default, that is, child processes will not share file descriptors.

question

Generally, in order to implement a multi-process and multi-threaded webserver, the service port fd must be set to inherit (set_inheritable), so that multiple processes can monitor a port (with SO_REUSEPORT)
The typical scenario is to use flask’s test server, here we write a piece of code to simulate.

 1
 2
 3
 4
 5
 6

 import socket, os
 server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
 server.bind(( '127.0.0.1' , 22222 ))
 server.set_inheritable( True )

 os.system( "python -c 'import time;time.sleep(1000)' " )

We can see all the file descriptors of these two processes through lsof -p {pid}
server process, you can see that the fd of the service port is 4

 1
 2
 3
 4
 5
 6
 7
 8
 9
 10
 11

 COMMAND PID FD TYPE DEVICE SIZE/OFF NODE NAME
 ptpython 6214 cwd DIR 253,0 4096 872946898 /
 ...
 ptpython 6214 0u CHR 136,13 0t0 16 /dev/pts/13
 ptpython 6214 1u CHR 136,13 0t0 16 /dev/pts/13
 ptpython 6214 2u CHR 136,13 0t0 16 /dev/pts/13
 ptpython 6214 3r CHR 1,9 0t0 2057 /dev/urandom
 ptpython 6214 4u sock 0,7 0t0 58345077 protocol: TCP
 ptpython 6214 5u a_inode 0,10 0 8627 [eventpoll]
 ptpython 6214 6u unix 0x0000000000000000 0t0 58368029 socket
 ptpython 6214 7u unix 0x0000000000000000 0t0 58368030 socket

The sleep child process also has a file descriptor of fd=4

 1
 2
 3
 4
 5
 6
 7

 COMMAND PID FD TYPE DEVICE SIZE/OFF NODE NAME
 python 18022 cwd DIR 253,0 4096 872946898/
 . . .
 python 18022 0u CHR 136,13 0t0 16 /dev/pts/13
 python 18022 1u CHR 136,13 0t0 16 /dev/pts/13
 python 18022 2u CHR 136,13 0t0 16 /dev/pts/13
 python 18022 4u sock 0,7 0t0 58345077 protocol: TCP

If the sleep process does not exit when the server process exits, the port corresponding to fd=4 is occupied, and the service cannot be started normally.

Solution

Manual cleanup

 1
 2
 3
 4
 5
 6
 7
 8

 import os
 import time

 os.system( f'lsof -p {os.getpid()} ' )
 os.closerange( 3 , 100 ) # This assumes that open file descriptors will not exceed 100
 time.sleep( 5 )
 os.system( f'lsof -p {os.getpid()} ' )
 # Execute the required business code later 

use close_fds

Use the subprocess library instead of os to start subprograms, and close redundant file descriptors with the close_fds parameter

 1
 2

 import subprocess
 subprocess.call( "python -c 'import time;time.sleep(1000)'" , shell= True , close_fds= True ) 

refer to

• https://docs.python.org/3/library/os.html#inheritance-of-file-descriptors
• https://docs.python.org/3/library/subprocess.html#subprocess.Popen
• https://stackoverflow.com/questions/2023608/check-what-files-are-open-in-python#answer-25069136

This article is reprinted from: http://ponder.work/2022/08/30/close-subprocess-opened-fd/
This site is for inclusion only, and the copyright belongs to the original author.