Threads in Processes with Python
You can configure and start new threading.Thread instances within new child multiprocessing.Process instances.
In this tutorial you will discover how to start threads in child processes in Python.
Let's get started.
Need To Start Threads in Child Processes
A process is a running instance of a computer program.
Every Python program is executed in a Process, which is a new instance of the Python interpreter. This process has the name MainProcess and has one thread used to execute the program instructions called the MainThread. Both processes and threads are created and managed by the underlying operating system.
Sometimes we may need to create new child processes in our program in order to execute code concurrently.
Python provides the ability to create and manage new processes via the multiprocessing.Process class.
You can learn more about multiprocessing in the tutorial:
In multiprocessing, we may need to start new threads in child processes.
For example, this is a common requirement when we need to perform many IO-bound tasks within each child process.
How can we start threads within processes in Python?
How to Start Threads in a Child Process
We can start new threads in child processes directly.
A new thread can be created via the threading.Thread class.
We can specify the function to execute in a new thread via the "target" argument.
For example:
...
# create a new thread
thread = threading.Thread(target=thread_task)Where, thread_task refers to a custom function we might define to run in a new thread.
# function to run in a new thread
def thread_task():
# ...
Once configured we can start the thread by calling the start() method.
For example:
...
# start the thread
thread.start()You can learn more about running functions in new threads in the tutorial:
We can then start new child processes to start new threads.
This can be achieved by defining a custom function that starts as many threads that are required to execute a given second custom function.
For example:
# function to run in a child process
def child_task():
# create a new thread
thread = threading.Thread(target=thread_task)
# start the thread
thread.start()
# ...
A new multiprocessing.Process instance can then be created and configured to execute the custom function to start threads and then start itself.
...
# configure a new child process
child = multiprocessing.Process(target=child_task)If you are new to running functions in a new child process, see the tutorial:
Now that we know how to start threads in child processes, let's look at a worked example.
Example of Starting Threads in Processes
We can explore how to start threads in child processes.
In this example we will start five child processes. Each child process will execute a task that involves starting three threads. Each thread will block for a random fraction of three seconds, then report a message. This means that there will be a total of five new child processes and 15 new threads running in the application, plus the main process and the six main threads running in each process.
First, we can define a function to execute in each thread.
The function will block for a random fraction of three seconds, then report that it is done including both the process name and the thread name.
The multiprocessing.current_process() function can be used to get the current process, and the "name" attribute can be used to access the name of the current process.
Similarly, the threading.current_thread() function can be used to get the current thread and the "name" attribute accessed to get the name of the current thread.
The thread_task() function below implements this.
# task executed by new threads
def thread_task():
# block for a moment
sleep(3 * random())
# report a message
process_name = current_process().name
thread_name = current_thread().name
print(f'>thread {thread_name} in process {process_name} done.', flush=True)
Next, we can define a function to be executed by each child process.
The function will first create a list of three threads. This can be achieved in a list comprehension.
The threads are then started and the child process blocks until all threads have finished.
Finally, the child process gets the name of the current process and reports that it is done.
The process_task() function below implements this.
# task executed by new processes
def process_task():
# configure new threads
threads = [Thread(target=thread_task) for _ in range(3)]
# start new threads
for thread in threads:
thread.start()
# wait for threads to finish
for thread in threads:
thread.join()
# report message
process_name = current_process().name
print(f'Process {process_name} done.', flush=True)
Finally, in the main process, we can create a list of five child processes. This can be achieved using a list comprehension.
The child processes are then started and the main process blocks until all child processes are finished.
The main process then reports that it is complete.
# protect the entry point
if __name__ == '__main__':
# configure child processes
processes = [Process(target=process_task) for _ in range(5)]
# start new processes
for child in processes:
child.start()
# wait for processes to finish
for child in processes:
child.join()
# report message
print('Main process done.', flush=True)
Tying this together, the complete example is listed below.
# SuperFastPython.com
# example of starting threads in child processes
from random import random
from time import sleep
from threading import Thread
from threading import current_thread
from multiprocessing import Process
from multiprocessing import current_process
# task executed by new threads
def thread_task():
# block for a moment
sleep(3 * random())
# report a message
process_name = current_process().name
thread_name = current_thread().name
print(f'>thread {thread_name} in process {process_name} done.', flush=True)
# task executed by new processes
def process_task():
# configure new threads
threads = [Thread(target=thread_task) for _ in range(3)]
# start new threads
for thread in threads:
thread.start()
# wait for threads to finish
for thread in threads:
thread.join()
# report message
process_name = current_process().name
print(f'Process {process_name} done.', flush=True)
# protect the entry point
if __name__ == '__main__':
# configure child processes
processes = [Process(target=process_task) for _ in range(5)]
# start new processes
for child in processes:
child.start()
# wait for processes to finish
for child in processes:
child.join()
# report message
print('Main process done.', flush=True)
Running the example first configures then starts five child processes.
The main process then blocks until all child processes finish.
Each child process then configures and starts three worker threads. The child processes then block until their own worker thread finishes.
Each worker thread blocks for a random fraction of three seconds, then reports a message.
The threads for a child process finish and the child process continues on and reports a message.
The child processes all finish, and the main process continues on and reports its final message.
This highlights how to start and manage new threads with new processes in Python.
Note, results will differ each time the code is run given the use of random numbers.
>thread Thread-2 in process Process-1 done.
>thread Thread-1 in process Process-4 done.
>thread Thread-3 in process Process-4 done.
>thread Thread-3 in process Process-3 done.
>thread Thread-3 in process Process-5 done.
>thread Thread-2 in process Process-2 done.
>thread Thread-2 in process Process-5 done.
>thread Thread-2 in process Process-3 done.
>thread Thread-1 in process Process-2 done.
>thread Thread-1 in process Process-5 done.
Process Process-5 done.
>thread Thread-3 in process Process-2 done.
Process Process-2 done.
>thread Thread-1 in process Process-1 done.
>thread Thread-1 in process Process-3 done.
Process Process-3 done.
>thread Thread-2 in process Process-4 done.
Process Process-4 done.
>thread Thread-3 in process Process-1 done.
Process Process-1 done.
Main process done.
Takeaways
You now know how to start threads in child processes in Python.