Last Updated on September 12, 2022
You can run a long-running monitoring task using a daemon thread and a while loop.
In this tutorial you will discover how to execute long-running tasks in the background in Python.
Let’s get started.
Need to Run a Long-Running Background Task
A thread is a thread of execution in a computer program.
Every Python program has at least one thread of execution called the main thread. Both processes and threads are created and managed by the underlying operating system.
Sometimes we may need to create additional threads in our program in order to execute code concurrently.
Python provides the ability to create and manage new threads via the threading module and the threading.Thread class.
You can learn more about Python threads in the guide:
In concurrent programming, we may need to run a long-running task in the background.
This may be for many reasons, such as:
- Monitor an external resource for change.
- Monitor program state for specific conditions.
- Monitor data for negative or favorable values.
Here, long-running means for the duration of the main program.
Background means that the task is not the main purpose of the application, but instead is a task that supports the main application.
How can we run long-running tasks in the background in Python?
Run loops using all CPUs, download your FREE book to learn how.
How to Run a Long-Running Background Task
You can run a long-running task in the background using a daemon thread.
A daemon thread is a background thread. Daemon is pronounced “dee-mon“, like the alternate spelling “demon“.
A thread may be configured to be a daemon or not, and most threads in concurrent programming, including the main thread, are non-daemon threads (not background threads) by default. The difference between daemon threads and non-daemon threads is that the process will exit if only daemon threads are running, whereas it cannot exit if at least one non-daemon thread is running.
As such, daemon threads are helpful for executing tasks in the background to support the non-daemon threads in an application.
If you are new to daemon threads, you can learn more about them here:
A Python threading.Thread instance can be configured to be a daemon thread.
We can configure a new thread to be a daemon thread by specifying the “daemon” argument to True in the constructor of the threading.Thread class.
For example:
|
1 2 3 |
... # create a new daemon thread thread = Thread(daemon=True, ...) |
We can configure a new daemon thread to execute a custom function that will perform a long-running task, such as monitor a resource or data.
For example we might define a new function named background_task().
|
1 2 3 |
# long-running background task def background_task(msg_queue): # ... |
Then, we can configure a new threading.Thread instance to execute this function via the “target” argument. The thread can be made daemon via the “daemon” argument and the thread can be given a meaningful name via the “name” argument.
|
1 2 3 |
... # create and start the daemon thread daemon = Thread(target=background_task, daemon=True, name='Monitor') |
Long-running means that the daemon thread will run for the duration of the main application.
This can be achieved with a while-loop that does not end.
|
1 2 3 4 |
... # run forever while True: # ... |
Each iteration of the loop, the task can check for the desirable state, such as a change to an external resource, program state or global variable, then take action.
In order to lessen the computation burden of this while-loop we can force the thread to sleep for a fraction of a second each iteration.
For example, we can configure the thread to sleep for 100 milliseconds each iteration, which means that the monitoring check would be performed about 10 times per second or 10hz.
|
1 2 3 |
... # block for a while sleep(0.1) |
Because the thread executing our task is a daemon thread, it will not prevent the application from exiting once the tasks of the main thread are finished.
Now that we know how to run a long-running task, let’s look at a worked example.
Example of a Long-Running Background Task
We can explore how to run a long-running background task in a daemon thread.
In this example, we will define a new function that will monitor program data via a global variable. This new task will be started in the main thread and run in the background while the main thread proceeds with its own tasks and modifies the shared data.
This is a simple monitoring task that could be expanded to monitor external resources such as a file or application specific data.
Firstly, we can define a new function named background_task() that will execute our background task.
|
1 2 3 |
# long-running background task def background_task(): # ... |
We can then correctly specify the global state of the shared global variable named “data” and store the current version for future comparison.
|
1 2 3 4 |
... global data # record the last seen value last_seen = data |
We can then start the while-loop that will run for the duration of the application.
|
1 2 3 |
# run forever while True: # ... |
Each iteration of the loop, we will check if the global variable has changed and if so take an action.
|
1 2 3 4 |
... # check for change if data != last_seen: # ... |
In this case, we will take the simple action of reporting the change, then updating the last seen version of the data so that the monitor can look for future changes.
|
1 2 3 4 5 |
... # report the change print(f'Monitor: data has changed to {data}') # update last seen last_seen = data |
Finally, each iteration, regardless of whether the global data changed or not, the thread will sleep for a fraction of a second. This delay can be chosen based on how responsive the monitor needs to be to changes within the application.
In this case, it will check for changes every 100 milliseconds, or about 10 times per second.
|
1 2 3 |
... # block for a while sleep(0.1) |
Tying this together, the complete background_task() function is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 |
# long-running background task def background_task(): global data # record the last seen value last_seen = data # run forever while True: # check for change if data != last_seen: # report the change print(f'Monitor: data has changed to {data}') # update last seen last_seen = data # block for a while sleep(0.1) |
Next, we can define the global data that will be monitored by the long-running background task.
|
1 2 3 |
... # global data data = 0 |
We can then create and start the background daemon thread to execute our background_task() function.
|
1 2 3 4 5 |
... # create and start the daemon thread print('Starting background task...') daemon = Thread(target=background_task, daemon=True, name='Monitor') daemon.start() |
The main thread will then carry on with the task of the application.
In this case, it will loop five times and each iteration it will block some random fraction of five seconds and then update the global state.
This will in turn cause the long-running monitor thread to detect the change and take action, logging a message automatically.
|
1 2 3 4 5 6 7 8 9 |
... # main thread is carrying on... print('Main thread is carrying on...') for _ in range(5): # block for a while value = random() * 5 sleep(value) # update the data variable data = value |
Tying this together, the complete example of a long-running background task is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 |
# SuperFastPython.com # example of a long-running daemon thread from time import sleep from random import random from threading import Thread # long-running background task def background_task(): global data # record the last seen value last_seen = data # run forever while True: # check for change if data != last_seen: # report the change print(f'Monitor: data has changed to {data}') # update last seen last_seen = data # block for a while sleep(0.1) # global data data = 0 # create and start the daemon thread print('Starting background task...') daemon = Thread(target=background_task, daemon=True, name='Monitor') daemon.start() # main thread is carrying on... print('Main thread is carrying on...') for _ in range(5): # block for a while value = random() * 5 sleep(value) # update the data variable data = value print('Main thread done.') |
Running the example first creates and starts the long-running background daemon thread.
The while-loop of the daemon thread executes and starts checking for changes to the global variable every 100 milliseconds.
The main thread then carries on executing and starts its loop, blocks for a random interval and changes the global variable.
The change to the global variable is noticed by the monitor thread, which in turn reports the change and keeps track of the change in order to notice future changes.
The main thread completes its loop and terminates the application, in turn abruptly terminating the daemon thread.
Note, your specific results will differ given the use of random numbers.
|
1 2 3 4 5 6 7 |
Starting background task... Main thread is carrying on... Monitor: data has changed to 3.205476019478387 Monitor: data has changed to 0.6740850729357128 Monitor: data has changed to 4.371193329005971 Monitor: data has changed to 0.7070589931520027 Main thread done. |
Free Python Threading Course
Download your FREE threading PDF cheat sheet and get BONUS access to my free 7-day crash course on the threading API.
Discover how to use the Python threading module including how to create and start new threads and how to use a mutex locks and semaphores
Further Reading
This section provides additional resources that you may find helpful.
Python Threading Books
- Python Threading Jump-Start, Jason Brownlee (my book!)
- Threading API Interview Questions
- Threading Module API Cheat Sheet
I also recommend specific chapters in the following books:
- Python Cookbook, David Beazley and Brian Jones, 2013.
- See: Chapter 12: Concurrency
- Effective Python, Brett Slatkin, 2019.
- See: Chapter 7: Concurrency and Parallelism
- Python in a Nutshell, Alex Martelli, et al., 2017.
- See: Chapter: 14: Threads and Processes
Guides
- Python Threading: The Complete Guide
- Python ThreadPoolExecutor: The Complete Guide
- Python ThreadPool: The Complete Guide
APIs
References
Overwhelmed by the python concurrency APIs?
Find relief, download my FREE Python Concurrency Mind Maps
Takeaways
You now know how to run long-running tasks in the background in Python.
Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.
Photo by Harley-Davidson on Unsplash
a very good talking + simple codes ,I fully understand at lease.thank you. also having something to asked …. if this kind of long terms running with 5 units, does its running fine?
Thanks.
Perhaps try it for your use case and see if it is a good fit.
Thank You for sharing.. appreciate.. Really help me a lot…
You’re very welcome!
Dear Jason, I’m currently trying to master concurrency in Python and see that you have wealth of info with clear explanations on this, thank you very much! Here are my 2 cents: updating “data” before going to sleep in the for loop will result in printing all 5 updates. Or did you do this on purpose to illustrate that as soon as main finishes the daemon is killed?
Cheers, Karel
Updating data before the sleep in the entry point does not generally change the program. We still get 5 prints. There could be (is) a race condition if the generated random value is less than 0.1. Is this what you meant? If so, it’s just an illustrative example with a mock task.
Am I overlooking something here, I count 4 lines reading “Monitor: data…”? So in this case it does change the output (not the program) when keeping all other settings as shown. In the example we get 4 lines due to change in data, the last one however, never shows since main sleeps, then changes data and finishes immediately… Is this what you call a race condition?
Ah, I see what you mean now, thanks.
Yes, the sleep in main is about simulating effort to generate the new data. It is okay if the monitor misses changes in this demo.