Asyncio tasks can suppress requests to be canceled and actively consume CancelledError exceptions.
Although consuming CancelledError exceptions is discouraged, it is possible to develop tasks that do this.
Nevertheless, there are cases where we may think that a CancelledError exception will be consumed, when in fact it is not and is allowed to propagate normally. Examples include awaiting subtasks in a canceled task using an asyncio.gather() configured to return exceptions, canceling a shielded task, and canceling a task that suppresses CancelledError exceptions before it has had a chance to run.
In this tutorial, you will discover how asyncio tasks can consume CancelledError exceptions, and cases where we expect them to consume exceptions and they do not.
Let’s get started.
What is Asyncio Task Cancellation?
Asyncio tasks can be canceled.
This can be achieved by calling the cancel() method on the asyncio.Task. This will request that the task be canceled as soon as possible and return True if the request was successful or False if it was not, e.g. the task is already done.
Request the Task to be cancelled. This arranges for a CancelledError exception to be thrown into the wrapped coroutine on the next cycle of the event loop.
— Coroutines and Tasks
For example:
|
1 2 3 4 |
... # cancel a task if task.cancel(): # request successful |
The next time that the target task that was canceled resumes, a CancelledError exception will be raised.
This exception will bubble up to the top level of the task and cause it to stop running.
You can learn more about the mechanics of cancellation a task in the tutorial:
You can also learn more about task cancellation best practices in the tutorial:
Run loops using all CPUs, download your FREE book to learn how.
Tasks Can Consume The CancelledError Exception
It is possible for tasks to not be canceled when required.
This can happen if the CancelledError exception raised within the target task is consumed.
Sometimes this is referred to as the asyncio task swallowing the CancelledError exception.
When this happens, the expectation or contract with the cancel() method on the task is broken and the asyncio program may not behave as expected.
Consuming CancelledError exceptions is discouraged in the asyncio API.
… suppressing cancellation completely is not common and is actively discouraged.
— Coroutines and Tasks
How can CancelledError exceptions be consumed?
What are situations when we think that CancelledError exceptions will be consumed and are in fact not?
How Can Tasks Consume CancelledError Exception
A CancelledError exception can be consumed by a task when it is canceled.
There are perhaps two ways that a task may consume a CancelledError exception, they are:
- Await all child tasks within a try-except.
- Await all child tasks using contextlib.suppress().
These are some of the same methods that can be used generally to suppress the CancelledError in a normal asyncio program.
You can learn more about suppressing CancelledError exceptions in the tutorial:
Let’s take a closer look at each in turn.
Consume CancelledError Exception With try-except
A task may consume a CancelledError exception using a try-except block and handling the CancelledError.
This could be used by wrapping each awaited subtask in the task body.
Alternatively, this can be achieved by wrapping the body of the task in a try-except block and handling the raised CancelledError.
For example:
|
1 2 3 4 5 6 |
async def task(): try: # body of the task ... except asyncio.CancelledError: pass |
Consume CancelledError Exception With contextlib.suppress()
A task may consume the CancelledError exception if it awaits subtasks using the contextlib.suppress() context manager.
This can be used to wrap each subtask one by one. It may also be used to wrap the entire body of the task.
For example:
|
1 2 3 4 5 |
async def task(): # await the cancelled task with contextlib.suppress(asyncio.CancelledError): # body of the task ... |
Free Python Asyncio Course
Download your FREE Asyncio PDF cheat sheet and get BONUS access to my free 7-day crash course on the Asyncio API.
Discover how to use the Python asyncio module including how to define, create, and run new coroutines and how to use non-blocking I/O.
Cases Where Tasks Fail to Consume CancelledError Exception
There are also a number of approaches where we might expect an asyncio Task to consume the CancelledError exception but does not. They are:
- Await all child tasks within an asyncio.gather().
- Await all child tasks within an asyncio.shield().
- Cancel target task while pending
Let’s take a closer look at these situations.
Fail to Consume With asyncio.gather()
We may expect a task may consume the CancelledError exception if it awaits subtasks using the asyncio.gather() function with the “return_exceptions” argument set to True.
For example:
|
1 2 3 |
... # suppress CancelledError exception _ = await asyncio.gather(task, return_exceptions=True) |
The expectation here is that the CancelledError exception will be handled by the asyncio.gather() function and returned, preventing the outer task from being canceled.
This is not the case.
A task that uses asyncio.gather(…, return_exceptions=True) to consume CancelledError exceptions for subtasks will still raise a CancelledError.
This is because of the asyncio.gather() task itself. It will then cancel any tasks on which it is waiting, and then propagate the CancelledError up the parent task causing it to cancel.
You can learn more about how to use the asyncio.gather() function in the tutorial:
Fail to Consume With asyncio.shield()
We may expect a task may consume the CancelledError exception if it awaits subtasks using the asyncio.shield() function
For example:
|
1 2 3 |
... # suppress CancelledError exception await asyncio.shield(task) |
The expectation here is that the CancelledError exception will be consumed by the asyncio.shield() function.
This is not the case.
The asyncio.shield() will protect the inner task from the CancelledError, allowing it to potentially continue to run.
The asyncio.shield() will act as though it was canceled and raise a CancelledError.
This will propagate back up to the surrounding task and cause it to be canceled.
You can learn more about asyncio.shield() in the tutorial:
Fail to Consume While Task is Pending
We might expect a coroutine that wraps the task body with a try-except or the contextlib.suppress() context manager to always consume the CancelledError.
For example:
|
1 2 3 4 5 6 |
async def task(): try: # body of the task ... except asyncio.CancelledError: pass |
Or:
|
1 2 3 4 5 |
async def task(): # await the cancelled task with contextlib.suppress(asyncio.CancelledError): # body of the task ... |
The expectation is that this task will always consume the CancelledError exception, no matter what.
This is not the case.
If the task is scheduled and not yet running, e.g. pending, then it can be canceled.
In this case, the asyncio event loop managing the task will respond to the request to cancel and raise a CancelledError, regardless of what is in the body of the task.
Now that we know that tasks can consume CancelledError exceptions, and cases where they will not, let’s look at some worked examples.
Overwhelmed by the python concurrency APIs?
Find relief, download my FREE Python Concurrency Mind Maps
Example of Task Consuming a CancelledError With try-except
We can explore an example of a task consuming a CancelledError exception using a try-except block.
In this example, we will define a coroutine that will sleep for a long time and then report a message. It will protect itself from cancellation by adding a try-except block to the body of the task and not propagating the CancelledError.
The work() coroutine below implements this.
|
1 2 3 4 5 6 7 8 9 10 |
# task that take a long time async def work(): try: # sleep a long time await asyncio.sleep(10) # report a final message print('Task sleep completed normally') except asyncio.CancelledError: # suppress CancelledError pass |
The main() coroutine will create and schedule the work() task and then wait a moment to allow it to run. It will then resume and cancel the task and await it before reporting a final message.
|
1 2 3 4 5 6 7 8 9 10 11 12 |
# main coroutine async def main(): # create and schedule the task task = asyncio.create_task(work()) # allow the task to start await asyncio.sleep(2) # cancel the task task.cancel() # await the task await task # report a final message print('Main is done') |
The expectation is that the work() task will consume the CancelledError exception, which will not propagate and the task will not cancel in the expected manner.
Finally, we can start the event loop and run the main() coroutine.
|
1 2 3 |
... # start the event loop asyncio.run(main()) |
Tying this together, the complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 |
# SuperFastPython.com # example of consuming CancelledError within the task import asyncio # task that take a long time async def work(): try: # sleep a long time await asyncio.sleep(10) # report a final message print('Task sleep completed normally') except asyncio.CancelledError: # suppress CancelledError pass # main coroutine async def main(): # create and schedule the task task = asyncio.create_task(work()) # allow the task to start await asyncio.sleep(2) # cancel the task task.cancel() # await the task await task # report a final message print('Main is done') # start the event loop asyncio.run(main()) |
Running the example first runs the event loop and executes the main() coroutine.
The main() coroutine runs and creates and schedules the work() coroutine as a background task. It then suspends for two seconds with a sleep.
The work() task runs and suspends for 10 seconds with a sleep.
The main() coroutine resumes and requests the work() task to cancel, then suspends and awaits the work() task to be done.
The work() task resumes and a CancelledError is raised in the sleep() coroutine. It is handled by the task and consumed. The CancelledError exception is not propagated. The work() task then terminates.
The main() coroutine resumes and reports a final message.
This example highlights how a task can be developed to consume the CancelledError exception.
|
1 |
Main is done |
Next, let’s look at an example of a task consuming a CancelledError exception using the contextlib.suppress() context manager.
Example of Task Consuming a CancelledError With contextlib.suppress()
We can explore an example of a task consuming a CancelledError exception using a contextlib.suppress() context manager.
In this case, we can update the above example so that the work() coroutine suppresses the CancelledError exception with the contextlib.suppress() context manager instead of a try-except block.
The body of the contextlib.suppress() context manager will contain the single subtask which is an await on the asyncio.sleep().
The updated work() coroutine with this change is listed below.
|
1 2 3 4 5 6 7 |
# task that take a long time async def work(): # sleep a long time with contextlib.suppress(asyncio.CancelledError): _ = await asyncio.sleep(10) # report a final message print('Task sleep completed normally') |
Again, the expectation is that the work() coroutine will consume the CancelledError exception, causing the cancellation request made by main() to be effectively ignored and not propagated.
Tying this together, the complete example is listed below
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 |
# SuperFastPython.com # example of consuming CancelledError with contextlib.suppress() import contextlib import asyncio # task that take a long time async def work(): # sleep a long time with contextlib.suppress(asyncio.CancelledError): _ = await asyncio.sleep(10) # report a final message print('Task sleep completed normally') # main coroutine async def main(): # create and schedule the task task = asyncio.create_task(work()) # allow the task to start await asyncio.sleep(2) # cancel the task task.cancel() # await the task await task # report a final message print('Main is done') # start the event loop asyncio.run(main()) |
Running the example first runs the event loop and executes the main() coroutine.
The main() coroutine runs and creates and schedules the work() coroutine as a background task. It then suspends for two seconds with a sleep.
The work() task runs and suspends for 10 seconds with a sleep.
The main() coroutine resumes and requests the work() task to cancel, then suspends and awaits the work() task to be done.
The work() task resumes and a CancelledError is raised in the sleep() coroutine. It is handled by the contextlib.suppress() context manager and consumed. The CancelledError exception is not propagated. The work() task resumes, reports a final message, and then terminates.
The main() coroutine resumes and reports a final message.
This example highlights how a task can be developed to consume the CancelledError exception using the contextlib.suppress() context manager.
|
1 2 |
Task sleep completed normally Main is done |
Next, let’s look at an example of how to use asyncio.gather() within a task that does not consume the CancelledError exception.
Example of Failing to Consume CancelledError With asyncio.gather()
We can explore an example of a task failing to consume a CancelledError exception using asyncio.gather().
In this case, we can update the work() coroutine so that the call to asyncio.sleep() is awaited within a call to asyncio.gather(), configured to return exceptions.
|
1 2 3 4 5 |
... # create sleep task coro = asyncio.sleep(10) # sleep a long time _ = await asyncio.gather(coro, return_exceptions=True) |
The naive expectation is that because asyncio.gather() is configured to return exceptions, a CancelledError propagated down to the asyncio.sleep() coroutine via the asyncio.gather() coroutine will be handled and returned.
The updated work() coroutine with this change is listed below.
|
1 2 3 4 5 6 7 8 |
# task that take a long time async def work(): # create sleep task coro = asyncio.sleep(10) # sleep a long time _ = await asyncio.gather(coro, return_exceptions=True) # report a final message print('Task sleep completed normally') |
Tying this together, the complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 |
# SuperFastPython.com # example of consuming CancelledError with asyncio.gather() import asyncio # task that take a long time async def work(): # create sleep task coro = asyncio.sleep(10) # sleep a long time _ = await asyncio.gather(coro, return_exceptions=True) # report a final message print('Task sleep completed normally') # main coroutine async def main(): # create and schedule the task task = asyncio.create_task(work()) # allow the task to start await asyncio.sleep(2) # cancel the task task.cancel() # await the task await task # report a final message print('Main is done') # start the event loop asyncio.run(main()) |
Running the example first runs the event loop and executes the main() coroutine.
The main() coroutine runs and creates and schedules the work() coroutine as a background task. It then suspends for two seconds with a sleep.
The work() task runs and suspends for 10 seconds with a sleep.
The main() coroutine resumes and requests the work() task to cancel, then suspends and awaits the work() task to be done.
The work() task resumes and a CancelledError is raised. The CancelledError is propagated down to the asyncio.gather() and from there down to the asyncio.sleep(). The asyncio.sleep() coroutine is canceled and the asyncio.gather() is canceled, then the CancelledError continues to propagate and terminates the work() task.
The main() coroutine resumes and the CancelledError exception is propagated and terminates the program.
This example highlights the expectation that the asyncio.gather() will consume the CancelledError exception when used within a task that is canceled was incorrect.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 |
Traceback (most recent call last): File "...", line 28, in <module> asyncio.run(main()) File ".../asyncio/runners.py", line 190, in run return runner.run(main) ^^^^^^^^^^^^^^^^ File ".../asyncio/runners.py", line 118, in run return self._loop.run_until_complete(task) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File ".../asyncio/base_events.py", line 653, in run_until_complete return future.result() ^^^^^^^^^^^^^^^ File "...", line 23, in main await task File "...", line 10, in work _ = await asyncio.gather(coro, return_exceptions=True) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "...", line 639, in sleep return await future ^^^^^^^^^^^^ asyncio.exceptions.CancelledError |
Next, let’s look at an example of how using asyncio.shield() on a task does not consume the CancelledError exception.
Example of Failing to Consume CancelledError With asyncio.shield()
We can explore an example of a task failing to consume a CancelledError exception using asyncio.shield().
In this case, we can update the main() coroutine so that the work() task is created and scheduled as normal, then wrapped in an asyncio.shield().
|
1 2 3 4 5 |
... # create and schedule the task task = asyncio.create_task((work())) # shield the task task = asyncio.shield(task) |
The naive expectation is that the asyncio.shield() will consume the CancelledError exception and allow the work() coroutine to continue to run.
The updated main() coroutine with this change is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 |
# main coroutine async def main(): # create and schedule the task task = asyncio.create_task((work())) # shield the task task = asyncio.shield(task) # allow the task to start await asyncio.sleep(2) # cancel the task task.cancel() # await the task await task # report a final message print('Main is done') |
Tying this together, the complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 |
# SuperFastPython.com # example of failing to consume CancelledError with asyncio.shield() import asyncio # task that take a long time async def work(): # sleep a long time await asyncio.sleep(10) # report a final message print('Task sleep completed normally') # main coroutine async def main(): # create and schedule the task task = asyncio.create_task((work())) # shield the task task = asyncio.shield(task) # allow the task to start await asyncio.sleep(2) # cancel the task task.cancel() # await the task await task # report a final message print('Main is done') # start the event loop asyncio.run(main()) |
Running the example first runs the event loop and executes the main() coroutine.
The main() coroutine runs and creates and schedules the work() coroutine as a background task. It then wraps the work() task in an asyncio.shield(). Next, the main() coroutine suspends for two seconds with a sleep.
The work() task runs and suspends for 10 seconds with a sleep.
The main() coroutine resumes and requests the work() task to cancel, then suspends and awaits the work() task to be done.
The asyncio.shield() consumes the CancelledError exception and the work() task is protected from the request to cancel.
Nevertheless, the asyncio.shield() responds to the cancellation request normally, raising the CancelledError.
The main() coroutine resumes and the CancelledError exception is propagated and terminates the program.
This example highlights the expectation that the asyncio.shield() will consume the CancelledError exception when used around a task that is canceled was incorrect. Instead, a shielded coroutine will continue to run, but the shield itself will raise and propagate the CancelledError normally.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 |
Traceback (most recent call last): File "...", line 28, in <module> asyncio.run(main()) File ".../asyncio/runners.py", line 190, in run return runner.run(main) ^^^^^^^^^^^^^^^^ File ".../asyncio/runners.py", line 118, in run return self._loop.run_until_complete(task) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File ".../asyncio/base_events.py", line 653, in run_until_complete return future.result() ^^^^^^^^^^^^^^^ File "...", line 23, in main await task asyncio.exceptions.CancelledError |
Next, let’s look at an example of how a task that actively suppresses CancelledError exceptions can still be canceled when in a pending state.
Example of Failing to Consume CancelledError When Pending
We can explore an example of a task failing to consume a CancelledError exception when it is scheduled and not yet running, e.g. pending.
In this case, we can update the above example so that the work() coroutine suppresses the CancelledError exception using a try-except block, as we did in the first example.
|
1 2 3 4 5 6 7 8 9 10 |
# task that take a long time async def work(): try: # sleep a long time await asyncio.sleep(10) # report a final message print('Task sleep completed normally') except asyncio.CancelledError: # suppress CancelledError pass |
We will then update the main() coroutine to schedule the task and then immediately cancel it.
The work() task will have no time to run before it is canceled.
The updated main() coroutine with this change is listed below.
|
1 2 3 4 5 6 7 8 9 10 |
# main coroutine async def main(): # create and schedule the task task = asyncio.create_task(work()) # cancel the task immediate task.cancel() # await the task await task # report a final message print('Main is done') |
Tying this together, the complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 |
# SuperFastPython.com # example of failing to consume CancelledError when pending import asyncio # task that take a long time async def work(): try: # sleep a long time await asyncio.sleep(10) # report a final message print('Task sleep completed normally') except asyncio.CancelledError: # suppress CancelledError pass # main coroutine async def main(): # create and schedule the task task = asyncio.create_task(work()) # cancel the task immediate task.cancel() # await the task await task # report a final message print('Main is done') # start the event loop asyncio.run(main()) |
Running the example first runs the event loop and executes the main() coroutine.
The main() coroutine runs and creates and schedules the work() coroutine as a background task.
The main() coroutine then immediately cancels the work() task and awaits it to be done.
The work() task has no opportunity to run.
The asyncio event loop cancels the work() task, raising a CancelledError, which terminates the work() task before it can run and propagates into the main() coroutine terminating the program.
This highlights how even though the body of a task may be developed to suppress a CancelledError exception, the task may still be canceled if the request to cancel is made before the body of the task is executed, e.g. the task is pending.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 |
Traceback (most recent call last): File "...", line 28, in <module> asyncio.run(main()) File "...", line 190, in run return runner.run(main) ^^^^^^^^^^^^^^^^ File ".../asyncio/runners.py", line 118, in run return self._loop.run_until_complete(task) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File ".../asyncio/base_events.py", line 653, in run_until_complete return future.result() ^^^^^^^^^^^^^^^ File "...", line 23, in main await task File "...", line 6, in work async def work(): asyncio.exceptions.CancelledError |
Further Reading
This section provides additional resources that you may find helpful.
Python Asyncio Books
- Python Asyncio Mastery, Jason Brownlee (my book!)
- Python Asyncio Jump-Start, Jason Brownlee.
- Python Asyncio Interview Questions, Jason Brownlee.
- Asyncio Module API Cheat Sheet
I also recommend the following books:
- Python Concurrency with asyncio, Matthew Fowler, 2022.
- Using Asyncio in Python, Caleb Hattingh, 2020.
- asyncio Recipes, Mohamed Mustapha Tahrioui, 2019.
Guides
APIs
- asyncio — Asynchronous I/O
- Asyncio Coroutines and Tasks
- Asyncio Streams
- Asyncio Subprocesses
- Asyncio Queues
- Asyncio Synchronization Primitives
References
Takeaways
You now know how asyncio tasks can consume CancelledError exceptions, and cases where we expect them to consume exceptions and they do not.
Did I make a mistake? See a typo?
I’m a simple humble human. Correct me, please!
Do you have any additional tips?
I’d love to hear about them!
Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.
Photo by Lance Asper on Unsplash
Do you have any questions?