Does the ThreadPool Stop Main From Exiting
The ThreadPool will be closed automatically by the Python garbage collector if required.
It will not prevent the program from exiting and the worker threads will not keep running if the main thread exits without closing the pool.
In this tutorial, we will explore what happens if you forget or are unable to close the ThreadPool in your program.
Let's get started.
Will The ThreadPool Stop The Program From Exiting?
The ThreadPool provides a pool of reusable workers for executing ad hoc tasks with thread-based concurrency.
An instance of the ThreadPool class can be created, specifying the number of workers to create, otherwise, a default number of workers will be created to match the number of logical CPUs in the system.
Once created, ad hoc tasks can be issued to the pool for execution via the apply() method. Multiple tasks may be executed in the book by calling the same function with different arguments via the map() method.
Tasks may be executed asynchronously in the pool via apply_async() and map_async(), allowing the caller to carry on with other tasks.
One concern with the ThreadPool is whether it will prevent the program from exiting if it is not explicitly closed.
For example, we may not close the ThreadPool correctly for many reasons, such as:
- We forget to call the close() or terminate() function.
- An error is raised that prevents a normal exit.
Sometimes in Python concurrency, a running thread or running child process can prevent the main process, and therefore the program from exiting normally.
Additionally, we may be concerned that if the pool is not closed properly, the worker threads will keep running if the main thread of the main process exits.
Will a running ThreadPool prevent the program from exiting normally (or will the program hang)?
ThreadPool Do Not Stop The Program From Exiting
A running ThreadPool does not stop the program from exiting.
This means that we can forget to close a ThreadPool and the main thread will exit normally.
For example:
# protect the entry point
if __name__ == '__main__':
# create a pool
pool = ThreadPool(4)
# ...
It also means that if an exception or error is raised and we are unable to close the pool normally, it will not prevent the program from exiting.
For example:
# protect the entry point
if __name__ == '__main__':
# create a pool
pool = ThreadPool(4)
# ...
# something bad happens
raise Exception('Something bad happened')
# close the pool
pool.close()
In both cases, the pool will be closed automatically by the garbage collector in the Python interpreter (parent process) and the worker threads will be closed and released.
Although this occurs, it should not be relied upon.
Note that it is not correct to rely on the garbage collector to destroy the pool as CPython does not assure that the finalizer of the pool will be called
-- multiprocessing — Process-based parallelism
Additionally, the worker threads are created as daemon threads.
This means that they are intended to run in the background and do not prevent the program from exiting.
You can learn more about daemon threads in the tutorial:
Now that we know that the ThreadPool will not stop the program from exiting and that the python garbage collector will close the pool for us if needed, let's look at a worked example.
Example of Not Closing a ThreadPool
We can explore the effect of not closing the ThreadPool.
In this example, we will create a ThreadPool with a fixed number of workers. We will then confirm that all worker threads are running and then attempt to exit the program.
Firstly, we can create a ThreadPool with four workers.
Don't worry if you have more or fewer CPU cores, as we will not be running any tasks.
...
# create a pool
pool = ThreadPool(4)Next, we can wait a moment for the pool to initialize completely. This is probably not required, but will likely avoid any possible race conditions in checking if the workers are running.
...
# wait a moment
sleep(0.5)We can then get a list of all active threads for the current process. This will include all workers in the ThreadPool.
This can be achieved via the threading.enumerate() function.
...
# report all running threads
for thread in threading.enumerate():
print(thread)
We will then attempt to exit the program normally, with the ThreadPool still running.
Tying this together, the complete example is listed below.
# SuperFastPython.com
# does the thread pool stop the program from exiting
from time import sleep
from multiprocessing.pool import ThreadPool
import threading
# protect the entry point
if __name__ == '__main__':
# create a pool
pool = ThreadPool(4)
# wait a moment
sleep(0.5)
# report all active threads
for thread in threading.enumerate():
print(thread)
# attempt to exit the main program without closing the pool
# ...
Running the example first creates a ThreadPool with four worker threads.
The main thread then blocks a moment to wait for the pool to initialize completely.
Next, a list of all running threads is retrieved and their details are reported.
We can see the four worker threads and three helper threads in the running ThreadPool.
We can notice that the worker threads and helper threads in the ThreadPool are daemon threads, meaning that they are intended to run in the background.
The main thread then ends normally, and the program exits.
The Python interpreter garbage collector in the process terminates the ThreadPool automatically.
<_MainThread(MainThread, started 4623429120)>
<DummyProcess(Thread-1, started daemon 123145483235328)>
<DummyProcess(Thread-2, started daemon 123145500024832)>
<DummyProcess(Thread-3, started daemon 123145516814336)>
<DummyProcess(Thread-4, started daemon 123145533603840)>
<Thread(Thread-5, started daemon 123145550393344)>
<Thread(Thread-6, started daemon 123145567182848)>
<Thread(Thread-7, started daemon 123145583972352)>
Takeaways
You now know what happens if you forget or are unable to close the ThreadPool.
If you enjoyed this tutorial, you will love my book: Python ThreadPool Jump-Start. It covers everything you need to master the topic with hands-on examples and clear explanations.