Why Asyncio Task Never Runs and Completes
You can develop an asyncio program that schedules background tasks, but then never gives them an opportunity to run or complete.
We can allow background tasks the opportunity to start running after they are scheduled by awaiting asyncio.sleep(0). We can also allow background tasks the opportunity to be completed without being canceled by awaiting them before the asyncio event loop exits and cancels them.
In this tutorial, you will discover why background asyncio tasks never run and how to fix them so they do run.
Let's get started.
Asyncio Coroutine or Task Never Runs
It is common to get a situation in asyncio where a coroutine never runs.
This can happen when the coroutine is first scheduled as a background task.
The asyncio program then proceeds normally and then at some point exits.
We will discover that the background task that we scheduled was never executed. Alternately, perhaps it started to execute but never finished.
Why are coroutines or background tasks never run?
Why Background Tasks Never Run
A background task can never run if the coroutine that scheduled it never awaits.
For example, we may have a coroutine that creates and schedules another coroutine as a background task:
...
# schedule the background task
_ = asyncio.create_task(work())It then proceeds with other work.
At some point the program ends and the background task has never run or may have started and never finished, even if it was short in duration.
The reason is that the coroutine that created and scheduled the background task did not await.
Because it did not await, the background task was never given an opportunity to execute.
To fix the problem, the calling coroutine must await and allow the background task to run.
How to Force Background Tasks To Start
The calling coroutine that created and scheduled the background task can force the new task to start.
This can be achieved by following the scheduling of the background task with an await for asyncio.sleep(0).
For example:
...
# schedule the background task
_ = asyncio.create_task(work())
# allow the background task to sleep
await asyncio.sleep(0)This will suspend the caller and allow all other scheduled coroutines and tasks an opportunity to run until their next point of suspension.
This is not a hack.
Instead, it is a pattern in asynchronous programming of suspending the caller and allowing other tasks to run.
You can learn more about the asyncio.sleep(0) pattern in the tutorial:
How to Force Background Tasks To Complete
The background task will never get a chance to complete.
This is because when the main coroutine exits, the event loop will exit. When the event loop exits, it will cancel all other tasks, including our background task.
You can learn more about this special ability of the main coroutine in the tutorial:
The calling coroutine can force the background task to be completed.
This can be achieved by awaiting the task directly.
We may want to do this at some later point in the program, such as at the end of the current coroutine.
The background task must first be assigned so that we have a reference to the asyncio.Task object.
For example:
...
# schedule the background task
task = asyncio.create_task(work())Then later in the program we can explicitly await it. This will ensure that it is given an opportunity to complete.
For example:
...
# allow the background task to complete
await taskIf we don't have access to the scheduled background tasks, we can loop over and await all tasks in the event loop, except the current task (self).
For example:
...
# get a set of all running tasks
all_tasks = asyncio.all_tasks()
# get the current tasks
current_task = asyncio.current_task()
# remove the current task from the list of all tasks
all_tasks.remove(current_task)
# suspend until all tasks are completed
await asyncio.wait(all_tasks)You can learn more about how to wait for all background tasks to complete in the tutorial:
Now that we know why background tasks may not run and how to fix the problem, let's explore some worked examples.
Example of Asyncio Task Never Runs
We can explore an example of a background task that is scheduled but never runs and is completed.
Firstly, we can define a simple background task that reports a message, sleeps for one second, and then prints a final message.
# background task
async def work():
# report a message
print('Work is starting')
# simulate work for a moment
await asyncio.sleep(1)
# report a message
print('Work is done')
Next, we can define a main coroutine that prints a message, schedules the work() coroutine as a background task, does some computational work, and then prints a final message.
# entry point coroutine
async def main():
# report a message
print('Main is starting')
# schedule the background task
_ = asyncio.create_task(work())
# do other things for a moment
results = [i*i for i in range(100000000)]
# report a message
print('Main is done')
Note that the background task is never given an opportunity to run because the main() coroutine does not await.
Finally, we can start the asyncio event loop.
...
# start the asyncio program
asyncio.run(main())Tying this together, the complete example is listed below.
# SuperFastPython.com
# example of a background task that never runs
import asyncio
# background task
async def work():
# report a message
print('Work is starting')
# simulate work for a moment
await asyncio.sleep(1)
# report a message
print('Work is done')
# entry point coroutine
async def main():
# report a message
print('Main is starting')
# schedule the background task
_ = asyncio.create_task(work())
# do other things for a moment
results = [i*i for i in range(100000000)]
# report a message
print('Main is done')
# start the asyncio program
asyncio.run(main())
Running the example first starts the asyncio event loop and runs the main() coroutine.
The main() coroutine runs and reports a message.
It then schedules the work() coroutine as a background task.
Next, it executes a list comprehension which takes a moment.
Finally, it prints a final message and terminates.
At no time while the main() coroutine was running did the work() task get an opportunity to run.
As the asyncio event loop is terminating it allows the background work() task to start running.
The work() task starts running and prints a message then suspends with a sleep.
The asyncio event loop then cancels the task and closes the program.
This highlights how a background task can be scheduled and not given an opportunity to run or complete.
Main is starting
Main is done
Work is startingNext, let's confirm that the background task was canceled by the asyncio event loop.
Example Confirming Background Task is Canceled
We can confirm that the background task was canceled by the asyncio event loop when the event loop was shut down.
In this case, we can update the above example to wrap the body of the work() task in a try-except and handle the CancelledError exception.
If this exception occurs, we can report a message and confirm that indeed the background task was canceled by the event loop.
The updated work() coroutine with this change is listed below.
# background task
async def work():
try:
# report a message
print('Work is starting')
# simulate work for a moment
await asyncio.sleep(1)
# report a message
print('Work is done')
except asyncio.CancelledError:
print('Cancelled')
Tying this together, the complete example is listed below.
# SuperFastPython.com
# example of a background task that is canceled by the event loop
import asyncio
# background task
async def work():
try:
# report a message
print('Work is starting')
# simulate work for a moment
await asyncio.sleep(1)
# report a message
print('Work is done')
except asyncio.CancelledError:
print('Cancelled')
# entry point coroutine
async def main():
# report a message
print('Main is starting')
# schedule the background task
_ = asyncio.create_task(work())
# do other things for a moment
results = [i*i for i in range(100000000)]
# report a message
print('Main is done')
# start the asyncio program
asyncio.run(main())
Running the example first starts the asyncio event loop and runs the main() coroutine.
The main() coroutine runs and reports a message.
It then schedules the work() coroutine as a background task.
Next, it executes a list comprehension which takes a moment.
Finally, it prints a final message and terminates.
At no time while the main() coroutine was running did the work() task get an opportunity to run.
As the asyncio event loop is terminating it allows the background work() task to start running.
The work() task starts running and prints a message then suspends with a sleep.
The asyncio event loop then cancels the task and closes the program.
A canceled message is printed, confirming that indeed the work() background task was canceled by the event loop.
This highlights that background tasks are canceled by the event loop when exiting.
Main is starting
Main is done
Work is starting
CancelledNext, let's explore how we can allow our background tasks an opportunity to run sooner.
Example of Allowing Background Task to Run
We can explore an example that allows the background task an opportunity to run sooner.
In this case, we can update the main() coroutine so that it awaits with an asyncio.sleep(0) immediately after scheduled the work() task.
For example:
...
# schedule the background task
task = asyncio.create_task(work())
# allow the background task to start
await asyncio.sleep(0)This will allow the background task to begin immediately.
The updated main() coroutine with this change is listed below.
# entry point coroutine
async def main():
# report a message
print('Main is starting')
# schedule the background task
task = asyncio.create_task(work())
# allow the background task to start
await asyncio.sleep(0)
# do other things for a moment
results = [i*i for i in range(100000000)]
# report a message
print('Main is done')
Tying this together, the complete example is listed below.
# SuperFastPython.com
# example of allowing a background task to run
import asyncio
# background task
async def work():
# report a message
print('Work is starting')
# simulate work for a moment
await asyncio.sleep(1)
# report a message
print('Work is done')
# entry point coroutine
async def main():
# report a message
print('Main is starting')
# schedule the background task
task = asyncio.create_task(work())
# allow the background task to start
await asyncio.sleep(0)
# do other things for a moment
results = [i*i for i in range(100000000)]
# report a message
print('Main is done')
# start the asyncio program
asyncio.run(main())
Running the example first starts the asyncio event loop and runs the main() coroutine.
The main() coroutine runs and reports a message.
It then schedules the work() coroutine as a background task. It then suspends with a sleep for zero seconds and allows the work() coroutine to run.
The work() task starts running and prints a message then suspends with a sleep.
The main() coroutine resumes and executes a list comprehension which takes a moment.
Finally, the main() coroutine prints a final message and terminates.
As the asyncio event loop is terminating it cancels the work() background task.
This highlights that we can allow the background task to start running immediately after being scheduled, but it is still canceled by the event loop.
Main is starting
Work is starting
Main is doneNext, let's explore how we can allow the background task to be completed.
Example of Allowing Background Task to Complete
We can explore how we can allow the background task to be completed.
In this case, we can update the main() coroutine so that the background task starts after it is scheduled with an asyncio.sleep(0) as we did previously, and then await the task explicitly at the end of the coroutine.
For example:
...
# allow background task to complete
await taskThis will allow the background task to be completed before the main coroutine exits.
The updated main() coroutine with this change is listed below.
# entry point coroutine
async def main():
# report a message
print('Main is starting')
# schedule the background task
task = asyncio.create_task(work())
# allow the background task to start
await asyncio.sleep(0)
# do other things for a moment
results = [i*i for i in range(100000000)]
# allow background task to complete
await task
# report a message
print('Main is done')
Tying this together, the complete example is listed below.
# SuperFastPython.com
# example of allowing a background task to complete
import asyncio
# background task
async def work():
# report a message
print('Work is starting')
# simulate work for a moment
await asyncio.sleep(1)
# report a message
print('Work is done')
# entry point coroutine
async def main():
# report a message
print('Main is starting')
# schedule the background task
task = asyncio.create_task(work())
# allow the background task to start
await asyncio.sleep(0)
# do other things for a moment
results = [i*i for i in range(100000000)]
# allow background task to complete
await task
# report a message
print('Main is done')
# start the asyncio program
asyncio.run(main())
Running the example first starts the asyncio event loop and runs the main() coroutine.
The main() coroutine runs and reports a message.
It then schedules the work() coroutine as a background task. It then suspends with a sleep for zero seconds and allows the work() coroutine to run.
The work() task starts running and prints a message then suspends with a sleep.
The main() coroutine resumes and executes a list comprehension which takes a moment.
Next, the main() coroutine suspends and explicitly awaits the background task.
The work() task resumes, prints a final message, and then terminates.
Finally, the main() coroutine prints a final message and terminates.
This highlights that we can allow the background task to start running immediately after being scheduled and also allow it to finish running before the event loop cancels it when the main coroutine is terminated.
Main is starting
Work is starting
Work is done
Main is doneTakeaways
You now know why background asyncio tasks never run and how to fix it so they do run.
If you enjoyed this tutorial, you will love my book: Python Asyncio Jump-Start. It covers everything you need to master the topic with hands-on examples and clear explanations.