You can develop an asyncio program that schedules background tasks, but then never gives them an opportunity to run or complete.
We can allow background tasks the opportunity to start running after they are scheduled by awaiting asyncio.sleep(0). We can also allow background tasks the opportunity to be completed without being canceled by awaiting them before the asyncio event loop exits and cancels them.
In this tutorial, you will discover why background asyncio tasks never run and how to fix them so they do run.
Let’s get started.
Asyncio Coroutine or Task Never Runs
It is common to get a situation in asyncio where a coroutine never runs.
This can happen when the coroutine is first scheduled as a background task.
The asyncio program then proceeds normally and then at some point exits.
We will discover that the background task that we scheduled was never executed. Alternately, perhaps it started to execute but never finished.
Why are coroutines or background tasks never run?
Run loops using all CPUs, download your FREE book to learn how.
Why Background Tasks Never Run
A background task can never run if the coroutine that scheduled it never awaits.
For example, we may have a coroutine that creates and schedules another coroutine as a background task:
|
1 2 3 |
... # schedule the background task _ = asyncio.create_task(work()) |
It then proceeds with other work.
At some point the program ends and the background task has never run or may have started and never finished, even if it was short in duration.
The reason is that the coroutine that created and scheduled the background task did not await.
Because it did not await, the background task was never given an opportunity to execute.
To fix the problem, the calling coroutine must await and allow the background task to run.
How to Force Background Tasks To Start
The calling coroutine that created and scheduled the background task can force the new task to start.
This can be achieved by following the scheduling of the background task with an await for asyncio.sleep(0).
For example:
|
1 2 3 4 5 |
... # schedule the background task _ = asyncio.create_task(work()) # allow the background task to sleep await asyncio.sleep(0) |
This will suspend the caller and allow all other scheduled coroutines and tasks an opportunity to run until their next point of suspension.
This is not a hack.
Instead, it is a pattern in asynchronous programming of suspending the caller and allowing other tasks to run.
You can learn more about the asyncio.sleep(0) pattern in the tutorial:
Free Python Asyncio Course
Download your FREE Asyncio PDF cheat sheet and get BONUS access to my free 7-day crash course on the Asyncio API.
Discover how to use the Python asyncio module including how to define, create, and run new coroutines and how to use non-blocking I/O.
How to Force Background Tasks To Complete
The background task will never get a chance to complete.
This is because when the main coroutine exits, the event loop will exit. When the event loop exits, it will cancel all other tasks, including our background task.
You can learn more about this special ability of the main coroutine in the tutorial:
The calling coroutine can force the background task to be completed.
This can be achieved by awaiting the task directly.
We may want to do this at some later point in the program, such as at the end of the current coroutine.
The background task must first be assigned so that we have a reference to the asyncio.Task object.
For example:
|
1 2 3 |
... # schedule the background task task = asyncio.create_task(work()) |
Then later in the program we can explicitly await it. This will ensure that it is given an opportunity to complete.
For example:
|
1 2 3 |
... # allow the background task to complete await task |
If we don’t have access to the scheduled background tasks, we can loop over and await all tasks in the event loop, except the current task (self).
For example:
|
1 2 3 4 5 6 7 8 9 |
... # get a set of all running tasks all_tasks = asyncio.all_tasks() # get the current tasks current_task = asyncio.current_task() # remove the current task from the list of all tasks all_tasks.remove(current_task) # suspend until all tasks are completed await asyncio.wait(all_tasks) |
You can learn more about how to wait for all background tasks to complete in the tutorial:
Now that we know why background tasks may not run and how to fix the problem, let’s explore some worked examples.
Overwhelmed by the python concurrency APIs?
Find relief, download my FREE Python Concurrency Mind Maps
Example of Asyncio Task Never Runs
We can explore an example of a background task that is scheduled but never runs and is completed.
Firstly, we can define a simple background task that reports a message, sleeps for one second, and then prints a final message.
|
1 2 3 4 5 6 7 8 |
# background task async def work(): # report a message print('Work is starting') # simulate work for a moment await asyncio.sleep(1) # report a message print('Work is done') |
Next, we can define a main coroutine that prints a message, schedules the work() coroutine as a background task, does some computational work, and then prints a final message.
|
1 2 3 4 5 6 7 8 9 10 |
# entry point coroutine async def main(): # report a message print('Main is starting') # schedule the background task _ = asyncio.create_task(work()) # do other things for a moment results = [i*i for i in range(100000000)] # report a message print('Main is done') |
Note that the background task is never given an opportunity to run because the main() coroutine does not await.
Finally, we can start the asyncio event loop.
|
1 2 3 |
... # start the asyncio program asyncio.run(main()) |
Tying this together, the complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 |
# SuperFastPython.com # example of a background task that never runs import asyncio # background task async def work(): # report a message print('Work is starting') # simulate work for a moment await asyncio.sleep(1) # report a message print('Work is done') # entry point coroutine async def main(): # report a message print('Main is starting') # schedule the background task _ = asyncio.create_task(work()) # do other things for a moment results = [i*i for i in range(100000000)] # report a message print('Main is done') # start the asyncio program asyncio.run(main()) |
Running the example first starts the asyncio event loop and runs the main() coroutine.
The main() coroutine runs and reports a message.
It then schedules the work() coroutine as a background task.
Next, it executes a list comprehension which takes a moment.
Finally, it prints a final message and terminates.
At no time while the main() coroutine was running did the work() task get an opportunity to run.
As the asyncio event loop is terminating it allows the background work() task to start running.
The work() task starts running and prints a message then suspends with a sleep.
The asyncio event loop then cancels the task and closes the program.
This highlights how a background task can be scheduled and not given an opportunity to run or complete.
|
1 2 3 |
Main is starting Main is done Work is starting |
Next, let’s confirm that the background task was canceled by the asyncio event loop.
Example Confirming Background Task is Canceled
We can confirm that the background task was canceled by the asyncio event loop when the event loop was shut down.
In this case, we can update the above example to wrap the body of the work() task in a try-except and handle the CancelledError exception.
If this exception occurs, we can report a message and confirm that indeed the background task was canceled by the event loop.
The updated work() coroutine with this change is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 |
# background task async def work(): try: # report a message print('Work is starting') # simulate work for a moment await asyncio.sleep(1) # report a message print('Work is done') except asyncio.CancelledError: print('Cancelled') |
Tying this together, the complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 |
# SuperFastPython.com # example of a background task that is canceled by the event loop import asyncio # background task async def work(): try: # report a message print('Work is starting') # simulate work for a moment await asyncio.sleep(1) # report a message print('Work is done') except asyncio.CancelledError: print('Cancelled') # entry point coroutine async def main(): # report a message print('Main is starting') # schedule the background task _ = asyncio.create_task(work()) # do other things for a moment results = [i*i for i in range(100000000)] # report a message print('Main is done') # start the asyncio program asyncio.run(main()) |
Running the example first starts the asyncio event loop and runs the main() coroutine.
The main() coroutine runs and reports a message.
It then schedules the work() coroutine as a background task.
Next, it executes a list comprehension which takes a moment.
Finally, it prints a final message and terminates.
At no time while the main() coroutine was running did the work() task get an opportunity to run.
As the asyncio event loop is terminating it allows the background work() task to start running.
The work() task starts running and prints a message then suspends with a sleep.
The asyncio event loop then cancels the task and closes the program.
A canceled message is printed, confirming that indeed the work() background task was canceled by the event loop.
This highlights that background tasks are canceled by the event loop when exiting.
|
1 2 3 4 |
Main is starting Main is done Work is starting Cancelled |
Next, let’s explore how we can allow our background tasks an opportunity to run sooner.
Example of Allowing Background Task to Run
We can explore an example that allows the background task an opportunity to run sooner.
In this case, we can update the main() coroutine so that it awaits with an asyncio.sleep(0) immediately after scheduled the work() task.
For example:
|
1 2 3 4 5 |
... # schedule the background task task = asyncio.create_task(work()) # allow the background task to start await asyncio.sleep(0) |
This will allow the background task to begin immediately.
The updated main() coroutine with this change is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 |
# entry point coroutine async def main(): # report a message print('Main is starting') # schedule the background task task = asyncio.create_task(work()) # allow the background task to start await asyncio.sleep(0) # do other things for a moment results = [i*i for i in range(100000000)] # report a message print('Main is done') |
Tying this together, the complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 |
# SuperFastPython.com # example of allowing a background task to run import asyncio # background task async def work(): # report a message print('Work is starting') # simulate work for a moment await asyncio.sleep(1) # report a message print('Work is done') # entry point coroutine async def main(): # report a message print('Main is starting') # schedule the background task task = asyncio.create_task(work()) # allow the background task to start await asyncio.sleep(0) # do other things for a moment results = [i*i for i in range(100000000)] # report a message print('Main is done') # start the asyncio program asyncio.run(main()) |
Running the example first starts the asyncio event loop and runs the main() coroutine.
The main() coroutine runs and reports a message.
It then schedules the work() coroutine as a background task. It then suspends with a sleep for zero seconds and allows the work() coroutine to run.
The work() task starts running and prints a message then suspends with a sleep.
The main() coroutine resumes and executes a list comprehension which takes a moment.
Finally, the main() coroutine prints a final message and terminates.
As the asyncio event loop is terminating it cancels the work() background task.
This highlights that we can allow the background task to start running immediately after being scheduled, but it is still canceled by the event loop.
|
1 2 3 |
Main is starting Work is starting Main is done |
Next, let’s explore how we can allow the background task to be completed.
Example of Allowing Background Task to Complete
We can explore how we can allow the background task to be completed.
In this case, we can update the main() coroutine so that the background task starts after it is scheduled with an asyncio.sleep(0) as we did previously, and then await the task explicitly at the end of the coroutine.
For example:
|
1 2 3 |
... # allow background task to complete await task |
This will allow the background task to be completed before the main coroutine exits.
The updated main() coroutine with this change is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 |
# entry point coroutine async def main(): # report a message print('Main is starting') # schedule the background task task = asyncio.create_task(work()) # allow the background task to start await asyncio.sleep(0) # do other things for a moment results = [i*i for i in range(100000000)] # allow background task to complete await task # report a message print('Main is done') |
Tying this together, the complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 |
# SuperFastPython.com # example of allowing a background task to complete import asyncio # background task async def work(): # report a message print('Work is starting') # simulate work for a moment await asyncio.sleep(1) # report a message print('Work is done') # entry point coroutine async def main(): # report a message print('Main is starting') # schedule the background task task = asyncio.create_task(work()) # allow the background task to start await asyncio.sleep(0) # do other things for a moment results = [i*i for i in range(100000000)] # allow background task to complete await task # report a message print('Main is done') # start the asyncio program asyncio.run(main()) |
Running the example first starts the asyncio event loop and runs the main() coroutine.
The main() coroutine runs and reports a message.
It then schedules the work() coroutine as a background task. It then suspends with a sleep for zero seconds and allows the work() coroutine to run.
The work() task starts running and prints a message then suspends with a sleep.
The main() coroutine resumes and executes a list comprehension which takes a moment.
Next, the main() coroutine suspends and explicitly awaits the background task.
The work() task resumes, prints a final message, and then terminates.
Finally, the main() coroutine prints a final message and terminates.
This highlights that we can allow the background task to start running immediately after being scheduled and also allow it to finish running before the event loop cancels it when the main coroutine is terminated.
|
1 2 3 4 |
Main is starting Work is starting Work is done Main is done |
Further Reading
This section provides additional resources that you may find helpful.
Python Asyncio Books
- Python Asyncio Mastery, Jason Brownlee (my book!)
- Python Asyncio Jump-Start, Jason Brownlee.
- Python Asyncio Interview Questions, Jason Brownlee.
- Asyncio Module API Cheat Sheet
I also recommend the following books:
- Python Concurrency with asyncio, Matthew Fowler, 2022.
- Using Asyncio in Python, Caleb Hattingh, 2020.
- asyncio Recipes, Mohamed Mustapha Tahrioui, 2019.
Guides
APIs
- asyncio — Asynchronous I/O
- Asyncio Coroutines and Tasks
- Asyncio Streams
- Asyncio Subprocesses
- Asyncio Queues
- Asyncio Synchronization Primitives
References
Takeaways
You now know why background asyncio tasks never run and how to fix it so they do run.
Did I make a mistake? See a typo?
I’m a simple humble human. Correct me, please!
Do you have any additional tips?
I’d love to hear about them!
Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.
Photo by Valdemaras Januška on Unsplash
Do you have any questions?