Last Updated on September 12, 2022
You can trigger events in a background thread using a daemon thread and a message queue.
In this tutorial you will discover how to run triggered tasks in the background in Python.
Let’s get started.
Need to Run a Triggered Background Task
A thread is a thread of execution in a computer program.
Every Python program has at least one thread of execution called the main thread. Both processes and threads are created and managed by the underlying operating system.
Sometimes we may need to create additional threads in our program in order to execute code concurrently.
Python provides the ability to create and manage new threads via the threading module and the threading.Thread class.
You can learn more about Python threads in the guude:
In concurrent programming, we may need to run a triggered task in the background.
This may be for many reasons, such as:
- Triggered by a change in application state.
- Requested by the application.
- In response to an external event.
Here, triggered means repeatedly, at any time, such as on demand.
Background means that the task is not the main purpose of the application, but instead is a task that supports the main application.
How can we run triggered tasks in the background in Python?
Run loops using all CPUs, download your FREE book to learn how.
How to Run a Triggered Background Task
You can run a triggered task in the background using a daemon thread and a thread-safe queue.
A daemon thread is a background thread. Daemon is pronounced “dee-mon“, like the alternate spelling “demon“.
A thread may be configured to be a daemon or not, and most threads in concurrent programming, including the main thread, are non-daemon threads (not background threads) by default. The difference between daemon threads and non-daemon threads is that the process will exit if only daemon threads are running, whereas it cannot exit if at least one non-daemon thread is running.
As such, daemon threads are helpful for executing tasks in the background to support the non-daemon threads in an application.
If you are new to daemon threads, you can learn more about them here:
A Python threading.Thread instance can be configured to be a daemon thread.
We can configure a new thread to be a daemon thread by specifying the “daemon” argument to True in the constructor of the threading.Thread class.
For example:
|
1 2 3 |
... # create a new daemon thread thread = Thread(daemon=True, ...) |
A queue is a data structure where messages can be placed with a call to put() and read with a call to get().
The queue.SimpleQueue provides a simple first-in-first-out (FIFO) queue that is also thread-safe. Thread-safe means that multiple different threads can concurrently call functions like get() and put() to retrieve and send messages with the queue without concurrency failure modes like race conditions or data corruption.
A message can be any Python object that is safe to share between threads.
For example:
|
1 2 3 |
... # place a message on the queue queue.put('Message') |
We can configure a new daemon thread to execute a custom function that will read messages from our queue.SimpleQueue and trigger an application-specific task.
For example we might define a new function named background_task() that takes the queue as an argument.
|
1 2 3 |
# task that runs when triggered def background_task(msg_queue): # ... |
Then configure a new threading.Thread instance to execute this function via the “target” keyword and pass the queue to the function via the “args” argument. The thread can be made daemon via the “daemon” argument and the thread can be given a meaningful name via the “name” argument.
|
1 2 3 4 |
... # create and start the daemon thread print('Starting background task...') daemon = Thread(target=background_task, args=(queue,), daemon=True, name='Background') |
Triggered means that the background task will execute each time a message is received via the queue.
This can be achieved with a while-loop that will block and wait for messages from the queue and once a message is received, will execute our desired task.
For example:
|
1 2 3 4 5 6 7 |
... # run forever while True: # get the next message message = msg_queue.get() # perform the task # ... |
Because the thread executing our task is a daemon thread, it will not prevent the application from exiting once the tasks of the main thread are finished.
Now that we know how to run triggered tasks in the background, let’s look at a worked example.
Example of Running a Triggered Background Task
We can explore how to run a triggered background task in a daemon thread.
In this example, we will define a new function that will execute tasks triggered by messages from a shared queue. This new task will be started in the main thread and run in the background while the main thread proceeds with its own tasks and pushes messages into the queue.
Firstly, we can define a new function named background_task() that will execute our background task.
The function will take one argument which is the shared queue.SimpleQueue instance from which triggering messages will be received.
|
1 2 3 |
# task that runs at a fixed interval def background_task(msg_queue): # ... |
We can then create a while-loop that will run for the duration of the program.
|
1 2 3 4 |
... # run forever while True: # ... |
Each iteration of the while loop will involve reading a message from the queue and then responding to it with an application-specific trigger.
In this case, the trigger will be to simply report it with a print() statement.
|
1 2 3 4 5 |
... # get the next message message = msg_queue.get() # log the message print(f'LOG: {message}') |
Note, that the call to get() on the queue will block until a message is available on the queue.
Tying this together, the complete background_task() function is listed below.
|
1 2 3 4 5 6 7 8 |
# task that runs when triggered def background_task(msg_queue): # run forever while True: # get the next message message = msg_queue.get() # log the message print(f'LOG: {message}') |
Next, we can create the message queue in the main thread.
|
1 2 3 |
... # create the message queue msg_queue = SimpleQueue() |
We can then create and start the background daemon thread to execute our background_task() function and pass it the shared message queue.
|
1 2 3 4 5 |
... # create and start the daemon thread print('Starting background task...') daemon = Thread(target=background_task, args=(msg_queue,), daemon=True, name='Background') daemon.start() |
The main thread will then carry on with the task of the application.
In this case, it will loop five times and each iteration it will block some random fraction of five seconds and then put() the result on the message queue. This will in turn trigger the daemon thread to log the message automatically.
|
1 2 3 4 5 6 7 8 9 10 |
... # main thread is carrying on... print('Main thread is carrying on...') for _ in range(5): # block for a while value = random() * 5 sleep(value) # log an application result msg_queue.put(f'computed {value}') print('Main thread done.') |
Tying this together, the complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 |
# SuperFastPython.com # example of a triggered daemon thread from time import sleep from random import random from threading import Thread from queue import SimpleQueue # task that runs when triggered def background_task(msg_queue): # run forever while True: # get the next message message = msg_queue.get() # log the message print(f'LOG: {message}') # create the message queue msg_queue = SimpleQueue() # create and start the daemon thread print('Starting background task...') daemon = Thread(target=background_task, args=(msg_queue,), daemon=True, name='Background') daemon.start() # main thread is carrying on... print('Main thread is carrying on...') for _ in range(5): # block for a while value = random() * 5 sleep(value) # log an application result msg_queue.put(f'computed {value}') print('Main thread done.') |
Running the example first creates and starts the background daemon thread.
The while-loop of the daemon thread starts and it blocks waiting on the message queue for messages to which to respond.
The main thread then carries on executing and starts its loop, blocks for a random interval and sends a message into the queue.
The arrival of each message in the queue triggers the background daemon thread which removes the messages and responds, in this case reporting the message.
The main thread completes its loop and terminates the application, in turn abruptly terminating the daemon thread.
Note, your specific results will differ given the use of random numbers.
|
1 2 3 4 5 6 7 |
Starting background task... Main thread is carrying on... LOG: computed 4.261181416228893 LOG: computed 4.641808920760989 LOG: computed 1.5761172059461042 LOG: computed 4.292315269561594 Main thread done. |
Free Python Threading Course
Download your FREE threading PDF cheat sheet and get BONUS access to my free 7-day crash course on the threading API.
Discover how to use the Python threading module including how to create and start new threads and how to use a mutex locks and semaphores
Further Reading
This section provides additional resources that you may find helpful.
Python Threading Books
- Python Threading Jump-Start, Jason Brownlee (my book!)
- Threading API Interview Questions
- Threading Module API Cheat Sheet
I also recommend specific chapters in the following books:
- Python Cookbook, David Beazley and Brian Jones, 2013.
- See: Chapter 12: Concurrency
- Effective Python, Brett Slatkin, 2019.
- See: Chapter 7: Concurrency and Parallelism
- Python in a Nutshell, Alex Martelli, et al., 2017.
- See: Chapter: 14: Threads and Processes
Guides
- Python Threading: The Complete Guide
- Python ThreadPoolExecutor: The Complete Guide
- Python ThreadPool: The Complete Guide
APIs
References
Overwhelmed by the python concurrency APIs?
Find relief, download my FREE Python Concurrency Mind Maps
Takeaways
You now know how to run triggered tasks in the background.
Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.
Photo by Harley-Davidson on Unsplash
Hello. In your tutos, except loop range(5), there are not a lot of difference between periodic and long-running.
For me long-running means running every minute or hour and every day.
Generally, periodic means on a schedule, e.g. every second, minute, hour, etc.
Long running could be something that is periodic or not. It could be monitoring a resource, it could be updating a display, etc.