Last Updated on October 29, 2022

Child worker threads are started automatically after creating an instance of the ThreadPool class.

In this tutorial, you will discover when the worker threads are created in the ThreadPool in Python.

Let’s get started.

ThreadPool Workers

The ThreadPool provides a pool of reusable workers for executing ad hoc tasks with thread-based concurrency.

An instance of the ThreadPool class can be created, specifying the number of workers to create, otherwise, a default number of workers will be created to match the number of logical CPUs in the system.

Once created, ad hoc tasks can be issued to the pool for execution via the apply() method. Multiple tasks may be executed in the book by calling the same function with different arguments via the map() method.

Tasks may be executed asynchronously in the pool via apply_async() and map_async() methods, allowing the caller to carry on with other tasks.

One concern with the ThreadPool is when exactly the worker threads used internally within the pool are created.

There are two main options for when child workers could be created:

• Immediately after an instance of the class is created.
• On-demand as tasks are issued to the pool.

We may need to know this for many reasons, such as:

• Will there be lag when the first task is issued if worker threads are created on demand?
• Will the pool need to be closed correctly if no tasks have been issued to the pool?

When are worker threads created in the ThreadPool?

When Are ThreadPool Workers Created

Workers in the ThreadPool are started immediately after the pool is created.

For example:

Moments after the class is created, it will internally create enough workers to populate the pool. In this case four.

This also applies when using the pool via the context manager interface.

For example:

Again, moments after the Pool instance is defined, it is populated with worker threads.

Now that we know when the worker threads are created in the ThreadPool, let’s look at a worked example.

Example of When Workers Are Created in the Pool

We can explore when the worker threads in the ThreadPool are created.

In this example, we will create a ThreadPool, wait a moment, then check if the caller thread has any running threads.

If it does have workers and the number of active threads matches the number of workers configured in the pool, it means that the child workers are created after the class is created.

Otherwise, it means that worker threads are created on demand.

This is an experimental approach to the question, an alternate approach would be to review the source code of the multiprocessing.pool.Pool class (the base class for the ThreadPool class).

Firstly, we can create a ThreadPool instance with four child worker threads. Don’t worry if you have more or fewer CPU cores, as we will not be running any tasks.

We will then wait a moment to allow any internal initialization by the pool to be completed.

Next, we can get access to all active threads for the current process and report their details.

This can be achieved via the threading.enumerate() module function.

This could be a list with one item, the main thread running the program. It could be a list with 5 or more threads, one for each worker, the main thread, and any other threads running.

Finally, we can close the pool now that we are finished with it.

Tying this together, the complete example is listed below.

Running the example first creates an instance of the ThreadPool.

The caller then blocks to allow the pool to start up completely. This is probably not necessary.

Next, a list of all running threads is retrieved and traversed, reporting the details of each.

In this case, we can see that the list of running threads contains many threads, including:

• the main thread, with the type _MainThread and name MainThread
• the four worker threads in the ThreadPool with the type DummyProcess and names Thread-1 to Thread-4.
• the three helper threads in the ThreadPool with the type Thread and names Thread-5 to Thread-7.

This confirms that worker threads are created after the class is created, and not on demand.

Finally, the ThreadPool is closed.

Further Reading

This section provides additional resources that you may find helpful.

Books

• Python ThreadPool Jump-Start, Jason Brownlee (my book!)
• Threading API Interview Questions
• ThreadPool PDF Cheat Sheet

I also recommend specific chapters from the following books:

• Python Cookbook, David Beazley and Brian Jones, 2013.
  • See: Chapter 12: Concurrency
• Effective Python, Brett Slatkin, 2019.
  • See: Chapter 7: Concurrency and Parallelism
• Python in a Nutshell, Alex Martelli, et al., 2017.
  • See: Chapter: 14: Threads and Processes

Guides

• Python ThreadPool: The Complete Guide
• Python Multiprocessing Pool: The Complete Guide
• Python ThreadPoolExecutor: The Complete Guide
• Python Threading: The Complete Guide

APIs

• multiprocessing - Process-based parallelism

References

• Thread (computing), Wikipedia.
• Process (computing), Wikipedia.
• Thread Pool, Wikipedia.

Takeaways

You now know when worker threads are created in the ThreadPool.

Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.

Photo by Faisal on Unsplash