Last Updated on November 14, 2023
Asyncio tasks can be canceled at any time.
This can cause a running task to stop mid-execution, which can cause problems if we expect a task or subtask to complete as an atomic operation.
Asyncio provides a way to shield tasks from cancellation via asyncio.shield(), making them directly immune from being canceled.
In this tutorial, you will discover how to shield asyncio tasks from cancellation.
After completing this tutorial, you will know:
- How to shield asyncio tasks from cancellation.
- How shields work and how we can work around them if we really need to.
- How to develop worked examples that use shields to protect tasks from cancellation.
Let’s get started.
What is Asyncio shield()
The asyncio.shield() function wraps an awaitable in Future that will absorb requests to be canceled.
Protect an awaitable object from being cancelled.
— Coroutines and Tasks
This means the shielded future can be passed around to tasks that may try to cancel it and the cancellation request will look like it was successful, except that the Task or coroutine that is being shielded will continue to run.
It may be useful in asyncio programs where some tasks can be canceled, but others, perhaps with a higher priority cannot.
It may also be useful in programs where some tasks can safely be canceled, such as those that were designed with asyncio in mind, whereas others cannot be safely terminated and therefore must be shielded from cancellation.
Now that we know what asyncio.shield() is, let’s look at how to use it.
Run loops using all CPUs, download your FREE book to learn how.
How to Use Asyncio shield()
The asyncio.shield() function will protect another Task or coroutine from being canceled.
It takes an awaitable as an argument and returns an asyncio.Future object.
The Future object can then be awaited directly or passed to another task or coroutine.
For example:
|
1 2 3 4 5 |
... # shield a task from cancellation shielded = asyncio.shield(task) # await the shielded task await shielded |
The returned Future can be canceled by calling the cancel() method.
If the inner task is running, the request will be reported as successful.
For example:
|
1 2 3 |
... # cancel a shielded task was_canceld = shielded.cancel() |
You can learn more about canceling asyncio tasks in the tutorial:
Any coroutines awaiting the Future object will raise an asyncio.CancelledError, which may need to be handled.
For example:
|
1 2 3 4 5 6 |
... try: # await the shielded task await asyncio.shield(task) except asyncio.CancelledError: # ... |
Importantly, the request for cancellation made on the Future object is not propagated to the inner task.
This means that the request for cancellation is absorbed by the shield.
For example:
|
1 2 3 4 5 6 7 |
... # create a task task = asyncio.create_task(coro()) # create a shield shield = asyncio.shield(task) # cancel the shield (does not cancel the task) shield.cancel() |
If a coroutine is provided to the asyncio.shield() function it is wrapped in an asyncio.Task() and scheduled immediately.
This means that the shield does not need to be awaited for the inner coroutine to run.
If aw is a coroutine it is automatically scheduled as a Task.
— Coroutines and Tasks
If the task that is being shielded is canceled, the cancellation request will be propagated up to the shield, which will also be canceled.
For example:
|
1 2 3 4 5 6 7 |
... # create a task task = asyncio.create_task(coro()) # create a shield shield = asyncio.shield(task) # cancel the task (also cancels the shield) task.cancel() |
Now that we know how to use the asyncio.shield() function, let’s look at some worked examples.
Example of Asyncio shield() for a Task
We can explore how to protect a task from cancellation using asyncio.shield().
In this example, we define a simple coroutine task that takes an integer argument, sleeps for a second, then returns the argument. The coroutine can then be created and scheduled as a Task.
We can define a second coroutine that takes a task, sleeps for a fraction of a second, then cancels the provided task.
In the main coroutine, we can then shield the first task and pass it to the second task, then await the shielded task.
The expectation is that the shield will be canceled and leave the inner task intact. The cancellation will disrupt the main coroutine. We can check the status of the inner task at the end of the program and we expect it to have been completed normally, regardless of the request to cancel made on the shield.
The complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 |
# SuperFastPython.com # example of using asyncio shield to protect a task from cancellation import asyncio # define a simple asynchronous async def simple_task(number): # block for a moment await asyncio.sleep(1) # return the argument return number # cancel the given task after a moment async def cancel_task(task): # block for a moment await asyncio.sleep(0.2) # cancel the task was_cancelled = task.cancel() print(f'cancelled: {was_cancelled}') # define a simple coroutine async def main(): # create the coroutine coro = simple_task(1) # create a task task = asyncio.create_task(coro) # created the shielded task shielded = asyncio.shield(task) # create the task to cancel the previous task asyncio.create_task(cancel_task(shielded)) # handle cancellation try: # await the shielded task result = await shielded # report the result print(f'>got: {result}') except asyncio.CancelledError: print('shielded was cancelled') # wait a moment await asyncio.sleep(1) # report the details of the tasks print(f'shielded: {shielded}') print(f'task: {task}') # start asyncio.run(main()) |
Running the example first creates the main() coroutine and uses it as the entry point into the application.
The task coroutine is created, then it is wrapped and scheduled in a Task.
The task is then shielded from cancellation.
The shielded task is then passed to the cancel_task() coroutine which is wrapped in a task and scheduled.
The main coroutine then awaits the shielded task, which expects a CancelledError exception.
The task runs for a moment then sleeps. The cancellation task runs for a moment, sleeps, resumes then cancels the shielded task. The request to cancel reports that it was successful.
This raises a CancelledError exception in the shielded Future, although not in the inner task.
The main() coroutine resumes and responds to the CancelledError exception, reporting a message. It then sleeps for a while longer.
The task resumes, finishes, and returns a value.
Finally, the main() coroutine resumes, and reports the status of the shielded future and the inner task. We can see that the shielded future is marked as canceled and yet the inner task is marked as finished normally and provides a return value.
This example highlights how a shield can be used to successfully protect an inner task from cancellation.
|
1 2 3 4 |
cancelled: True shielded was cancelled shielded: <Future cancelled> task: <Task finished name='Task-2' coro=<simple_task() done, defined at ...> result=1> |
Next, let’s look at shielding a coroutine, instead of a task, from cancellation.
Free Python Asyncio Course
Download your FREE Asyncio PDF cheat sheet and get BONUS access to my free 7-day crash course on the Asyncio API.
Discover how to use the Python asyncio module including how to define, create, and run new coroutines and how to use non-blocking I/O.
Example of Asyncio shield for a Coroutine
We can explore how to protect a coroutine from cancellation using asyncio.shield().
In this example, we update the above example to shield a coroutine directly, instead of a task.
The expectation is that the coroutine will be wrapped in a task immediately and scheduled for execution. The request for cancellation made on the shield will protect the inner coroutine from cancellation as it does for tasks.
At the end of the program, we will locate the asyncio.Task associated with the shielded coroutine and report its status. The expectation is that it will have finished normally and was not canceled.
The complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 |
# SuperFastPython.com # example of using asyncio shield to protect a coroutine from cancellation import asyncio # define a simple asynchronous async def simple_task(number): # block for a moment await asyncio.sleep(1) # return the argument return number # cancel the given task after a moment async def cancel_task(task): # block for a moment await asyncio.sleep(0.2) # cancel the task was_cancelled = task.cancel() print(f'cancelled: {was_cancelled}') # define a simple coroutine async def main(): # create the coroutine coro = simple_task(1) # created the shielded task shielded = asyncio.shield(coro) # create the task to cancel the previous task asyncio.create_task(cancel_task(shielded)) # handle cancellation try: # await the shielded task result = await shielded # report the result print(f'>got: {result}') except asyncio.CancelledError: print('shielded was cancelled') # get all tasks tasks = asyncio.all_tasks() # wait a moment await asyncio.sleep(1) # report the details of the tasks print(f'shielded: {shielded}') # report the task for the coroutine for task in tasks: if task.get_coro() is coro: print(f'task: {task}') # start asyncio.run(main()) |
Running the example first creates the main() coroutine and uses it as the entry point into the application.
The task coroutine is created. It is then shielded from cancellation. Internally this wraps the coroutine in a Task object and schedules it for execution. We will retrieve this Task object later.
The shielded task is then passed to the cancel_task() coroutine which is wrapped in a task and scheduled.
The main coroutine then awaits the shielded task, which expects a CancelledError exception.
The task runs for a moment then sleeps. The cancellation task runs for a moment, sleeps, resumes, then cancels the shielded task. The request to cancel reports that it was successful.
This raises a CancelledError exception in the shielded Future, although not in the inner task or coroutine.
The main() coroutine resumes and responds to the CancelledError exception, reporting a message. It then sleeps for a while longer.
The inner task resumes, finishes, and returns a value.
Finally, the main() coroutine resumes, reporting the status of the shielded future. It then locates the Task associated with the inner coroutine and reports its status.
We can see that the shielded future is marked as canceled and yet the inner task for the shielded coroutine is marked as finished normally and provides a return value.
This example highlights how a shield can be used to successfully protect an inner coroutine from cancellation, and that the shield() function will create an asyncio.Task object for a provided coroutine.
|
1 2 3 4 |
cancelled: True shielded was cancelled shielded: <Future cancelled> task: <Task finished name='Task-2' coro=<simple_task() done, defined at ...> result=1> |
Next, let’s look at what happens when we cancel the task with a shield.
Overwhelmed by the python concurrency APIs?
Find relief, download my FREE Python Concurrency Mind Maps
Example of Canceling Shielded Task
We can explore what happens when the inner shielded task is canceled.
In this example, we will pass the inner task to the cancellation coroutine.
The expectation is that the inner task will be canceled and that this request for cancellation will be propagated out to the shield and then impact the main coroutine.
The complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 |
# SuperFastPython.com # example of canceling the task directly inside the shield import asyncio # define a simple asynchronous async def simple_task(number): # block for a moment await asyncio.sleep(1) # return the argument return number # cancel the given task after a moment async def cancel_task(task): # block for a moment await asyncio.sleep(0.2) # cancel the task was_cancelled = task.cancel() print(f'cancelled: {was_cancelled}') # define a simple coroutine async def main(): # create the coroutine coro = simple_task(1) # create a task task = asyncio.create_task(coro) # created the shielded task shielded = asyncio.shield(task) # create the task to cancel the previous task asyncio.create_task(cancel_task(task)) # handle cancellation try: # await the shielded task result = await shielded # report the result print(f'>got: {result}') except asyncio.CancelledError: print('shielded was cancelled') # wait a moment await asyncio.sleep(1) # report the details of the tasks print(f'shielded: {shielded}') print(f'task: {task}') # start asyncio.run(main()) |
Running the example first creates the main() coroutine and uses it as the entry point into the application.
The task coroutine is created, then it is wrapped and scheduled in a Task.
The task is then shielded from cancellation.
The task itself, not the shielded task, is then passed to the cancel_task() coroutine that is wrapped in a task and scheduled.
The main coroutine then awaits the shielded task, which expects a CancelledError exception.
The task runs for a moment then sleeps. The cancellation task runs for a moment, sleeps, resumes then cancels the task directly. The request to cancel reports that it was successful.
This raises a CancelledError exception in the task itself which cancels. The CancelledError exception is then raised in the shielded Future object, which also cancels.
The main() coroutine resumes and responds to the CancelledError exception, reporting a message. It then sleeps for a while longer.
Finally, the main() coroutine resumes, and reports the status of the shielded future and the inner task.
We can see that both the shielded Future object and the inner Task are marked as canceled.
This example highlights that although a Task can be shielded, it can still be canceled directly.
|
1 2 3 4 |
cancelled: True shielded was cancelled shielded: <Future cancelled> task: <Task cancelled name='Task-2' coro=<simple_task() done, defined at ...>> |
Example of Asyncio shield() with wait_for()
We can explore what happens to a shielded task that is canceled by a call to wait_for() after a timeout.
In this example, we shield the task from cancellation as before. In this case, we pass the shielded Future to a call to the asyncio.wait_for() function.
This function will wait for a task to complete with a timeout. If the timeout elapses before the task is complete, it is canceled.
You can learn more about the wait_for() function in the tutorial:
The expectation is that the shielded future will be canceled after the timeout by the wait_for() function, although this will not impact the internal task.
The complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 |
# SuperFastPython.com # example of using wait_for() with a shielded task import asyncio # define a simple asynchronous async def simple_task(number): # block for a moment await asyncio.sleep(1) # return the argument return number # define a simple coroutine async def main(): # create the coroutine coro = simple_task(1) # create a task task = asyncio.create_task(coro) # created the shielded task shielded = asyncio.shield(task) # execute shielded task with a timeout try: # await the shielded task result = await asyncio.wait_for(shielded, timeout=0.2) # report the result print(f'>got: {result}') except asyncio.CancelledError: print('wait_for was cancelled') except asyncio.TimeoutError: print('timed out waiting for result') # wait a moment await asyncio.sleep(1) # report the details of the tasks print(f'shielded: {shielded}') print(f'task: {task}') # start asyncio.run(main()) |
Running the example first creates the main() coroutine and uses it as the entry point into the application.
The task coroutine is created, then it is wrapped and scheduled in a Task.
The task is then shielded from cancellation.
The shielded task is then passed to the wait_for() function and a timeout of a fraction of a second is used. This call expects a possible CancelledError exception if the shield is canceled and a TimeoutError if the task is canceled with a timeout.
The main coroutine then awaits the wait_for() task.
The task runs for a moment then sleeps. The timeout in the wait_for() task elapses and cancels the shielded Future.
This raises a TimeoutError exception in the shielded Future, although not in the inner task.
The main() coroutine resumes and responds to the TimeoutError exception, reporting a message. It then sleeps for a while longer.
The task resumes, finishes, and returns a value.
Finally, the main() coroutine resumes, and reports the status of the shielded future and the inner task. We can see that the shielded future is marked as canceled and yet the inner task is marked as finished normally and provides a return value.
This example highlights how the shield can be used to protect a time from cancellation due to a timeout when using the asyncio.wait_for() function.
|
1 2 3 |
timed out waiting for result shielded: <Future cancelled> task: <Task finished name='Task-2' coro=<simple_task() done, defined at ...> result=1> |
Common Questions
This section considers common questions related to shielding tasks from cancellation.
Do you have any questions about shielding tasks from cancellation?
Ask your questions in the comments below and I will do my best to answer them and may add them to this section.
Can a Task Be Shielded?
Yes.
Can a Coroutine Be Shielded?
Yes.
A coroutine passed to shield() will be wrapped in an asyncio.Task and scheduled immediately.
The Future returned from shield() does not need to be awaited in order for the provided coroutine to be executed.
How Can We Wait For a Shielded Task To Complete?
We can wait for a shielded task to complete by keeping a reference to the asyncio.Task object that was shielded and waiting for it directly.
Alternatively, if a coroutine was provided to the shield() function, then the associated asyncio.Task can be located from asyncio.all_tasks() and awaited.
How Can We Get The Result from a Canceled Shielded Task?
We can get the result from a canceled shielded task by keeping a reference to the asyncio.Task object that was shielded and called the result() method in order to get the return value.
Alternatively, if a coroutine was provided to the shield() function, then the associated asyncio.Task can be located from asyncio.all_tasks() and the result() method can be called on it.
What Happens if The Shielded Task Itself is Canceled?
If the inner task that is being shielded is canceled, it will cancel the external Future that wraps it.
This means that cancellations are propagated up from the inner task to the outer shield task.
Can the Inner Task Still Be Canceled?
Yes.
If the shielded inner asyncio.Task object is canceled directly, then the task will raise an asyncio.CancelledError exception and can cancel the task.
Further Reading
This section provides additional resources that you may find helpful.
Python Asyncio Books
- Python Asyncio Mastery, Jason Brownlee (my book!)
- Python Asyncio Jump-Start, Jason Brownlee.
- Python Asyncio Interview Questions, Jason Brownlee.
- Asyncio Module API Cheat Sheet
I also recommend the following books:
- Python Concurrency with asyncio, Matthew Fowler, 2022.
- Using Asyncio in Python, Caleb Hattingh, 2020.
- asyncio Recipes, Mohamed Mustapha Tahrioui, 2019.
Guides
APIs
- asyncio — Asynchronous I/O
- Asyncio Coroutines and Tasks
- Asyncio Streams
- Asyncio Subprocesses
- Asyncio Queues
- Asyncio Synchronization Primitives
References
Takeaways
You now know how to shield asyncio tasks from cancellation in Python.
Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.
Photo by Aaron Huber on Unsplash
Do you have any questions?