Original link: https://mazhuang.org/2023/04/11/after-using-threadpool/
Recently, when some friends asked me to check the problem, there were two related to the thread pool. In the end, it took some time to find out the problem and make a record for future reference for myself and others who need it.
1. Asynchronous to synchronous
Phenomenon:
There is a method that just submits a task to the thread pool after being requested, and then returns immediately, but judging from the traceId of the log, occasionally the method and the task are executed in the same thread, and the interface takes a long time.
Analysis process:
This is actually a knowledge point: when there are no idle threads in the thread pool and the task queue is full, how will the newly submitted tasks be processed?
You can look at the TheadPoolExecutor class, which has several predefined strategies (implements RejectedExecutionHandler):
- CallerRunsPolicy
- Abort Policy
- Discard Policy
- DiscardOldestPolicy
Combined with their names and comments, you can see that they correspond to:
- Schedule threads to perform tasks by themselves; (the exception is that the thread pool is shut down and the task is discarded)
- Ignore the task and throw an exception; (default)
- Discard the task without raising an exception;
- Discard the oldest unprocessed task in the queue, and then try to schedule a new task; (same exception)
In addition, you can also define your own policies as needed.
In our scenario, the RejectedExecutionHandler used by this thread pool is CallerRunsPolicy, so the reason is found.
solution:
Because the main requirement in the scenario is that the interface should return quickly without losing tasks, then it would be more appropriate to use a message queue in this case, so here, the task submitted to the thread pool is changed to send a message to the message queue.
2. The disappearing task
Phenomenon:
As can be seen from the log, a task is submitted to the thread pool, but no record of the task execution can be found.
Analysis process:
The first is to suspect that the task is discarded or ignored. It is confirmed that the RejectExecutionHandler of the thread pool is using the default AbortPolicy. If it is ignored, an exception will be thrown, but no exception record can be found in the log.
That is to say, it successfully entered the task queue, but it was not executed, where did it go?
After thinking hard, I wonder if the application has been killed? Check the scrolling records of the container in the K8S console. Sure enough, around the time when the task was submitted, the application version was released——solve the case.
solution:
Two ideas are offered:
- On the premise of ensuring that tasks are logically idempotent, tasks are scheduled through message queues, database record task status + retry mechanism, etc.;
- The container goes offline gracefully, and can only be killed after confirming that the requests and tasks being processed are completed.
This article is transferred from: https://mazhuang.org/2023/04/11/after-using-threadpool/
This site is only for collection, and the copyright belongs to the original author.