Asyncio CancelledError Can Be Consumed

January 23, 2024 Python Asyncio

Asyncio tasks can suppress requests to be canceled and actively consume CancelledError exceptions.

Although consuming CancelledError exceptions is discouraged, it is possible to develop tasks that do this.

Nevertheless, there are cases where we may think that a CancelledError exception will be consumed, when in fact it is not and is allowed to propagate normally. Examples include awaiting subtasks in a canceled task using an asyncio.gather() configured to return exceptions, canceling a shielded task, and canceling a task that suppresses CancelledError exceptions before it has had a chance to run.

In this tutorial, you will discover how asyncio tasks can consume CancelledError exceptions, and cases where we expect them to consume exceptions and they do not.

Let's get started.

What is Asyncio Task Cancellation?

Asyncio tasks can be canceled.

This can be achieved by calling the cancel() method on the asyncio.Task. This will request that the task be canceled as soon as possible and return True if the request was successful or False if it was not, e.g. the task is already done.

Request the Task to be cancelled. This arranges for a CancelledError exception to be thrown into the wrapped coroutine on the next cycle of the event loop.
-- Coroutines and Tasks

For example:

...
# cancel a task
if task.cancel():
	# request successful

The next time that the target task that was canceled resumes, a CancelledError exception will be raised.

This exception will bubble up to the top level of the task and cause it to stop running.

You can learn more about the mechanics of cancellation a task in the tutorial:

How to Cancel an Asyncio Task

You can also learn more about task cancellation best practices in the tutorial:

Asyncio Task Cancellation Best Practices

Tasks Can Consume The CancelledError Exception

It is possible for tasks to not be canceled when required.

This can happen if the CancelledError exception raised within the target task is consumed.

Sometimes this is referred to as the asyncio task swallowing the CancelledError exception.

When this happens, the expectation or contract with the cancel() method on the task is broken and the asyncio program may not behave as expected.

Consuming CancelledError exceptions is discouraged in the asyncio API.

... suppressing cancellation completely is not common and is actively discouraged.
-- Coroutines and Tasks

How can CancelledError exceptions be consumed?

What are situations when we think that CancelledError exceptions will be consumed and are in fact not?

How Can Tasks Consume CancelledError Exception

A CancelledError exception can be consumed by a task when it is canceled.

There are perhaps two ways that a task may consume a CancelledError exception, they are:

Await all child tasks within a try-except.
Await all child tasks using contextlib.suppress().

These are some of the same methods that can be used generally to suppress the CancelledError in a normal asyncio program.

You can learn more about suppressing CancelledError exceptions in the tutorial:

4 Ways to Suppress Asyncio CancelledError

Let's take a closer look at each in turn.

Consume CancelledError Exception With try-except

A task may consume a CancelledError exception using a try-except block and handling the CancelledError.

This could be used by wrapping each awaited subtask in the task body.

Alternatively, this can be achieved by wrapping the body of the task in a try-except block and handling the raised CancelledError.

For example:

async def task():
	try:
		# body of the task
		...
	except asyncio.CancelledError:
		pass

Consume CancelledError Exception With contextlib.suppress()

A task may consume the CancelledError exception if it awaits subtasks using the contextlib.suppress() context manager.

This can be used to wrap each subtask one by one. It may also be used to wrap the entire body of the task.

For example:

async def task():
	# await the cancelled task
	with contextlib.suppress(asyncio.CancelledError):
		# body of the task
		...

Cases Where Tasks Fail to Consume CancelledError Exception

There are also a number of approaches where we might expect an asyncio Task to consume the CancelledError exception but does not. They are:

Await all child tasks within an asyncio.gather().
Await all child tasks within an asyncio.shield().
Cancel target task while pending

Let's take a closer look at these situations.

Fail to Consume With asyncio.gather()

We may expect a task may consume the CancelledError exception if it awaits subtasks using the asyncio.gather() function with the "return_exceptions" argument set to True.

For example:

...
# suppress CancelledError exception
_ = await asyncio.gather(task, return_exceptions=True)

The expectation here is that the CancelledError exception will be handled by the asyncio.gather() function and returned, preventing the outer task from being canceled.

This is not the case.

A task that uses asyncio.gather(..., return_exceptions=True) to consume CancelledError exceptions for subtasks will still raise a CancelledError.

This is because of the asyncio.gather() task itself. It will then cancel any tasks on which it is waiting, and then propagate the CancelledError up the parent task causing it to cancel.

You can learn more about how to use the asyncio.gather() function in the tutorial:

How to Use asyncio.gather() in Python

Fail to Consume With asyncio.shield()

We may expect a task may consume the CancelledError exception if it awaits subtasks using the asyncio.shield() function

For example:

...
# suppress CancelledError exception
await asyncio.shield(task)

The expectation here is that the CancelledError exception will be consumed by the asyncio.shield() function.

This is not the case.

The asyncio.shield() will protect the inner task from the CancelledError, allowing it to potentially continue to run.

The asyncio.shield() will act as though it was canceled and raise a CancelledError.

This will propagate back up to the surrounding task and cause it to be canceled.

You can learn more about asyncio.shield() in the tutorial:

Asyncio Shield From Cancellation

Fail to Consume While Task is Pending

We might expect a coroutine that wraps the task body with a try-except or the contextlib.suppress() context manager to always consume the CancelledError.

For example:

async def task():
	try:
		# body of the task
		...
	except asyncio.CancelledError:
		pass

Or:

async def task():
	# await the cancelled task
	with contextlib.suppress(asyncio.CancelledError):
		# body of the task
		...

The expectation is that this task will always consume the CancelledError exception, no matter what.

This is not the case.

If the task is scheduled and not yet running, e.g. pending, then it can be canceled.

In this case, the asyncio event loop managing the task will respond to the request to cancel and raise a CancelledError, regardless of what is in the body of the task.

Now that we know that tasks can consume CancelledError exceptions, and cases where they will not, let's look at some worked examples.

Example of Task Consuming a CancelledError With try-except

We can explore an example of a task consuming a CancelledError exception using a try-except block.

In this example, we will define a coroutine that will sleep for a long time and then report a message. It will protect itself from cancellation by adding a try-except block to the body of the task and not propagating the CancelledError.

The work() coroutine below implements this.

# task that take a long time
async def work():
    try:
        # sleep a long time
        await asyncio.sleep(10)
        # report a final message
        print('Task sleep completed normally')
    except asyncio.CancelledError:
        # suppress CancelledError
        pass

The main() coroutine will create and schedule the work() task and then wait a moment to allow it to run. It will then resume and cancel the task and await it before reporting a final message.

# main coroutine
async def main():
    # create and schedule the task
    task = asyncio.create_task(work())
    # allow the task to start
    await asyncio.sleep(2)
    # cancel the task
    task.cancel()
    # await the task
    await task
    # report a final message
    print('Main is done')

The expectation is that the work() task will consume the CancelledError exception, which will not propagate and the task will not cancel in the expected manner.

Finally, we can start the event loop and run the main() coroutine.

...
# start the event loop
asyncio.run(main())

Tying this together, the complete example is listed below.

# SuperFastPython.com
# example of consuming CancelledError within the task
import asyncio

# task that take a long time
async def work():
    try:
        # sleep a long time
        await asyncio.sleep(10)
        # report a final message
        print('Task sleep completed normally')
    except asyncio.CancelledError:
        # suppress CancelledError
        pass

# main coroutine
async def main():
    # create and schedule the task
    task = asyncio.create_task(work())
    # allow the task to start
    await asyncio.sleep(2)
    # cancel the task
    task.cancel()
    # await the task
    await task
    # report a final message
    print('Main is done')

# start the event loop
asyncio.run(main())

Running the example first runs the event loop and executes the main() coroutine.

The main() coroutine runs and creates and schedules the work() coroutine as a background task. It then suspends for two seconds with a sleep.

The work() task runs and suspends for 10 seconds with a sleep.

The main() coroutine resumes and requests the work() task to cancel, then suspends and awaits the work() task to be done.

The work() task resumes and a CancelledError is raised in the sleep() coroutine. It is handled by the task and consumed. The CancelledError exception is not propagated. The work() task then terminates.

The main() coroutine resumes and reports a final message.

This example highlights how a task can be developed to consume the CancelledError exception.

Main is done

Next, let's look at an example of a task consuming a CancelledError exception using the contextlib.suppress() context manager.

Example of Task Consuming a CancelledError With contextlib.suppress()

We can explore an example of a task consuming a CancelledError exception using a contextlib.suppress() context manager.

In this case, we can update the above example so that the work() coroutine suppresses the CancelledError exception with the contextlib.suppress() context manager instead of a try-except block.

The body of the contextlib.suppress() context manager will contain the single subtask which is an await on the asyncio.sleep().

The updated work() coroutine with this change is listed below.

# task that take a long time
async def work():
    # sleep a long time
    with contextlib.suppress(asyncio.CancelledError):
        _ = await asyncio.sleep(10)
    # report a final message
    print('Task sleep completed normally')

Again, the expectation is that the work() coroutine will consume the CancelledError exception, causing the cancellation request made by main() to be effectively ignored and not propagated.

Tying this together, the complete example is listed below

# SuperFastPython.com
# example of consuming CancelledError with contextlib.suppress()
import contextlib
import asyncio

# task that take a long time
async def work():
    # sleep a long time
    with contextlib.suppress(asyncio.CancelledError):
        _ = await asyncio.sleep(10)
    # report a final message
    print('Task sleep completed normally')

# main coroutine
async def main():
    # create and schedule the task
    task = asyncio.create_task(work())
    # allow the task to start
    await asyncio.sleep(2)
    # cancel the task
    task.cancel()
    # await the task
    await task
    # report a final message
    print('Main is done')

# start the event loop
asyncio.run(main())

Running the example first runs the event loop and executes the main() coroutine.

The main() coroutine runs and creates and schedules the work() coroutine as a background task. It then suspends for two seconds with a sleep.

The work() task runs and suspends for 10 seconds with a sleep.

The main() coroutine resumes and requests the work() task to cancel, then suspends and awaits the work() task to be done.

The work() task resumes and a CancelledError is raised in the sleep() coroutine. It is handled by the contextlib.suppress() context manager and consumed. The CancelledError exception is not propagated. The work() task resumes, reports a final message, and then terminates.

The main() coroutine resumes and reports a final message.

This example highlights how a task can be developed to consume the CancelledError exception using the contextlib.suppress() context manager.

Task sleep completed normally
Main is done

Next, let's look at an example of how to use asyncio.gather() within a task that does not consume the CancelledError exception.

Example of Failing to Consume CancelledError With asyncio.gather()

We can explore an example of a task failing to consume a CancelledError exception using asyncio.gather().

In this case, we can update the work() coroutine so that the call to asyncio.sleep() is awaited within a call to asyncio.gather(), configured to return exceptions.

...
# create sleep task
coro = asyncio.sleep(10)
# sleep a long time
_ = await asyncio.gather(coro, return_exceptions=True)

The naive expectation is that because asyncio.gather() is configured to return exceptions, a CancelledError propagated down to the asyncio.sleep() coroutine via the asyncio.gather() coroutine will be handled and returned.

The updated work() coroutine with this change is listed below.

# task that take a long time
async def work():
    # create sleep task
    coro = asyncio.sleep(10)
    # sleep a long time
    _ = await asyncio.gather(coro, return_exceptions=True)
    # report a final message
    print('Task sleep completed normally')

Tying this together, the complete example is listed below.

# SuperFastPython.com
# example of consuming CancelledError with asyncio.gather()
import asyncio

# task that take a long time
async def work():
    # create sleep task
    coro = asyncio.sleep(10)
    # sleep a long time
    _ = await asyncio.gather(coro, return_exceptions=True)
    # report a final message
    print('Task sleep completed normally')

# main coroutine
async def main():
    # create and schedule the task
    task = asyncio.create_task(work())
    # allow the task to start
    await asyncio.sleep(2)
    # cancel the task
    task.cancel()
    # await the task
    await task
    # report a final message
    print('Main is done')

# start the event loop
asyncio.run(main())

Running the example first runs the event loop and executes the main() coroutine.

The main() coroutine runs and creates and schedules the work() coroutine as a background task. It then suspends for two seconds with a sleep.

The work() task runs and suspends for 10 seconds with a sleep.

The main() coroutine resumes and requests the work() task to cancel, then suspends and awaits the work() task to be done.

The work() task resumes and a CancelledError is raised. The CancelledError is propagated down to the asyncio.gather() and from there down to the asyncio.sleep(). The asyncio.sleep() coroutine is canceled and the asyncio.gather() is canceled, then the CancelledError continues to propagate and terminates the work() task.

The main() coroutine resumes and the CancelledError exception is propagated and terminates the program.

This example highlights the expectation that the asyncio.gather() will consume the CancelledError exception when used within a task that is canceled was incorrect.

Traceback (most recent call last):
  File "...", line 28, in <module>
    asyncio.run(main())
  File ".../asyncio/runners.py", line 190, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File ".../asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../asyncio/base_events.py", line 653, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "...", line 23, in main
    await task
  File "...", line 10, in work
    _ = await asyncio.gather(coro, return_exceptions=True)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "...", line 639, in sleep
    return await future
           ^^^^^^^^^^^^
asyncio.exceptions.CancelledError

Next, let's look at an example of how using asyncio.shield() on a task does not consume the CancelledError exception.

Example of Failing to Consume CancelledError With asyncio.shield()

We can explore an example of a task failing to consume a CancelledError exception using asyncio.shield().

In this case, we can update the main() coroutine so that the work() task is created and scheduled as normal, then wrapped in an asyncio.shield().

...
# create and schedule the task
task = asyncio.create_task((work()))
# shield the task
task = asyncio.shield(task)

The naive expectation is that the asyncio.shield() will consume the CancelledError exception and allow the work() coroutine to continue to run.

The updated main() coroutine with this change is listed below.

# main coroutine
async def main():
    # create and schedule the task
    task = asyncio.create_task((work()))
    # shield the task
    task = asyncio.shield(task)
    # allow the task to start
    await asyncio.sleep(2)
    # cancel the task
    task.cancel()
    # await the task
    await task
    # report a final message
    print('Main is done')

Tying this together, the complete example is listed below.

# SuperFastPython.com
# example of failing to consume CancelledError with asyncio.shield()
import asyncio

# task that take a long time
async def work():
    # sleep a long time
    await asyncio.sleep(10)
    # report a final message
    print('Task sleep completed normally')

# main coroutine
async def main():
    # create and schedule the task
    task = asyncio.create_task((work()))
    # shield the task
    task = asyncio.shield(task)
    # allow the task to start
    await asyncio.sleep(2)
    # cancel the task
    task.cancel()
    # await the task
    await task
    # report a final message
    print('Main is done')

# start the event loop
asyncio.run(main())

Running the example first runs the event loop and executes the main() coroutine.

The main() coroutine runs and creates and schedules the work() coroutine as a background task. It then wraps the work() task in an asyncio.shield(). Next, the main() coroutine suspends for two seconds with a sleep.

The work() task runs and suspends for 10 seconds with a sleep.

The main() coroutine resumes and requests the work() task to cancel, then suspends and awaits the work() task to be done.

The asyncio.shield() consumes the CancelledError exception and the work() task is protected from the request to cancel.

Nevertheless, the asyncio.shield() responds to the cancellation request normally, raising the CancelledError.

The main() coroutine resumes and the CancelledError exception is propagated and terminates the program.

This example highlights the expectation that the asyncio.shield() will consume the CancelledError exception when used around a task that is canceled was incorrect. Instead, a shielded coroutine will continue to run, but the shield itself will raise and propagate the CancelledError normally.

Traceback (most recent call last):
  File "...", line 28, in <module>
    asyncio.run(main())
  File ".../asyncio/runners.py", line 190, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File ".../asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../asyncio/base_events.py", line 653, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "...", line 23, in main
    await task
asyncio.exceptions.CancelledError

Next, let's look at an example of how a task that actively suppresses CancelledError exceptions can still be canceled when in a pending state.

Example of Failing to Consume CancelledError When Pending

We can explore an example of a task failing to consume a CancelledError exception when it is scheduled and not yet running, e.g. pending.

In this case, we can update the above example so that the work() coroutine suppresses the CancelledError exception using a try-except block, as we did in the first example.

# task that take a long time
async def work():
    try:
        # sleep a long time
        await asyncio.sleep(10)
        # report a final message
        print('Task sleep completed normally')
    except asyncio.CancelledError:
        # suppress CancelledError
        pass

We will then update the main() coroutine to schedule the task and then immediately cancel it.

The work() task will have no time to run before it is canceled.

The updated main() coroutine with this change is listed below.

# main coroutine
async def main():
    # create and schedule the task
    task = asyncio.create_task(work())
    # cancel the task immediate
    task.cancel()
    # await the task
    await task
    # report a final message
    print('Main is done')

Tying this together, the complete example is listed below.

# SuperFastPython.com
# example of failing to consume CancelledError when pending
import asyncio

# task that take a long time
async def work():
    try:
        # sleep a long time
        await asyncio.sleep(10)
        # report a final message
        print('Task sleep completed normally')
    except asyncio.CancelledError:
        # suppress CancelledError
        pass

# main coroutine
async def main():
    # create and schedule the task
    task = asyncio.create_task(work())
    # cancel the task immediate
    task.cancel()
    # await the task
    await task
    # report a final message
    print('Main is done')

# start the event loop
asyncio.run(main())

Running the example first runs the event loop and executes the main() coroutine.

The main() coroutine runs and creates and schedules the work() coroutine as a background task.

The main() coroutine then immediately cancels the work() task and awaits it to be done.

The work() task has no opportunity to run.

The asyncio event loop cancels the work() task, raising a CancelledError, which terminates the work() task before it can run and propagates into the main() coroutine terminating the program.

This highlights how even though the body of a task may be developed to suppress a CancelledError exception, the task may still be canceled if the request to cancel is made before the body of the task is executed, e.g. the task is pending.

Traceback (most recent call last):
  File "...", line 28, in <module>
    asyncio.run(main())
  File "...", line 190, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File ".../asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../asyncio/base_events.py", line 653, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "...", line 23, in main
    await task
  File "...", line 6, in work
    async def work():
asyncio.exceptions.CancelledError

Takeaways

You now know how asyncio tasks can consume CancelledError exceptions, and cases where we expect them to consume exceptions and they do not.