One skill a day: the wrong method doubles the code. How do Requests retry correctly?

Original link: https://www.kingname.info/2023/06/11/retry-in-requests/

Programmers are a group that needs continuous learning. If you find that the code you write now is no different from the code you wrote 5 years ago, it means you have fallen behind.

When we do Python development, we often use some third-party libraries, which have continued to add new functions for many years. But I found that many students will not use new functions at all when using these third-party libraries. Their code is the same as it was a few years ago.

For example, use Request to initiate an HTTP request. When the request fails, no matter what the reason is, retry up to 3 times in place. Many people mainly have the following three ways of writing to try again.

common old way

Use third-party libraries

Such students will use some third-party libraries that specialize in retrying, such as tenacity. See my article for details: Tenacity——Exception Retry has never been easier

Manually write decorators

This kind of students will use decorators, so they usually write decorators by hand to reuse them, for example:

 1
 2
 3
 4
 5
 6
 7
 8
 9
 10
 11
 12
 13
 14
 15

 def retry ( func ):
 def wrap ( *args, **kwargs ):
 for _ in range ( 3 ):
 try :
 result = func(*args, **kwargs)
 return result
 except Exception as e:
 print ( 'An error occurred, try again' )
 return {}
 return wrap


 @retry
 def make_request ( url ):
 print ( 'The following is the relevant code for initiating the request' ) 

repeated for loop

There are also some students who write code in a wild way:

 1
 2
 3
 4
 5
 6
 7
 8

 def login ():
 for i in range ( 10 ): # retry 10 times
 try :
 resp = requests.get( 'XX URL' )
 return resp. json()
 except Exception as e:
 print ( f'request error, retry {i} th time' )
 continue

Such students basically do not reuse code. In the code, if requests are made to N urls, they will write codes like the above in N places.

new method

Although I am talking about a new method here, this method should have been available at least 9 years ago. It’s just that fewer people use it online. We can use HTTPAdapter that comes with requests to implement automatic retry. This method can be used when we don’t care what the specific error is and just need to retry mechanically:

 1
 2
 3
 4
 5
 6
 7
 8
 9
 10
 11
 12

 import requests
 from requests.adapters import HTTPAdapter, Retry

 session = requests. Session()
 retries = Retry(total= 3 , backoff_factor= 1 )
 session. mount( 'http://' , HTTPAdapter(max_retries=retries))
 session. mount( 'https://' , HTTPAdapter(max_retries=retries))

 # Next, all requests initiated using session will be retried up to 3 times by default
 session.get( 'http://httpbin.org/delay/5' , timeout= 2 )
 session.get( 'https://www.kingname.info' )
 ...

This article is transferred from: https://www.kingname.info/2023/06/11/retry-in-requests/
This site is only for collection, and the copyright belongs to the original author.