We can run asyncio.gather() in the background by not awaiting the call to asyncio.gather().
The asyncio.gather() returns an asyncio.Future that does not have to be awaited. It is scheduled for execution in the asyncio event loop, along with all coroutines provided to the gather. Therefore, the caller is free to proceed with other activities and the gather will automatically be executed in the background.
In this tutorial, you will discover how to run asyncio.gather() in the background.
Let’s get started.
Need to Run asyncio.gather() in the Background
The asyncio.gather() function allows us to run multiple coroutines or tasks concurrently.
Coroutines can be provided as positional arguments to asyncio.gather() and a list of return values is returned, for example:
|
1 2 3 |
... # execute coroutines concurrently results = await asyncio.gather(coro1(), coro2()) |
Alternately, we can create a list of coroutines or tasks and unpack them as positional arguments using the star operator (*), for example:
|
1 2 3 4 5 |
... # creates a list of coroutine objects coros = [coro(i) for i in range(100)] # execute coroutines concurrently results = await asyncio.gather(*coros) |
You can learn more about how to use asyncio.gather() in the tutorial:
We may need to run asyncio.gather() in the background.
For example, we may need to execute many concurrent tasks or coroutines, but not block the flow of execution in the current coroutine.
How can we run asyncio.gather() in the background?
Run loops using all CPUs, download your FREE book to learn how.
How to Run asyncio.gather() in the Background
We can run asyncio.gather() in the background directly by not awaiting the awaitable that is returned.
The asyncio.gather() function will automatically schedule the provided coroutines as tasks that will run in the background.
If any awaitable in aws is a coroutine, it is automatically scheduled as a Task.
— Coroutines and Tasks
The awaitable returned from asyncio.gather() is an asyncio.Future which is like an asyncio.Task. It too will be automatically scheduled for independent execution.
Therefore, we can issue the asyncio.gather() and proceed with other activities, without suspending the current coroutine, and all issued coroutines in the call to gather() will execute and complete in the background.
For example:
|
1 2 3 4 5 6 |
... # create many coroutines coros = [task_coro(i) for i in range(10)] # get the future for the gather future = asyncio.gather(*coros) # proceed with other work... |
If we need to retrieve the return values from all issued coroutines, we can call the result() method on the asyncio.Future object that is returned.
For example:
|
1 2 3 |
... # get the results results = future.result() |
Now that we know how to execute an asyncio.gather() call in the background, let’s look at some worked examples.
Example of asyncio.gather() with Many Coroutines
Before we look at how we might run asyncio.gather() in the background, let’s look at an example of running asyncio.gather() normally, in the foreground.
In this case, we can define a custom coroutine that suspends a moment, reports a message, and returns a result.
|
1 2 3 4 5 6 7 8 |
# coroutine used for a task async def task_coro(value): # sleep for a moment await asyncio.sleep(2) # report a message print(f'>{value} done') # return the result return value * 100 |
In the main coroutine, we can create a list of 10 coroutines with arguments from 0 to 9, execute them concurrently with asyncio.gather(), then report the results.
The main() coroutine below implements this.
|
1 2 3 4 5 6 7 8 9 10 11 12 |
# coroutine used for the entry point async def main(): # report a message print('Main starting') # create many coroutines coros = [task_coro(i) for i in range(10)] # run the tasks in the background results = await asyncio.gather(*coros) # report results print(results) # report a message print('Main done') |
Tying this together, the complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 |
# SuperFastPython.com # example of running an asyncio gather with many tasks import asyncio # coroutine used for a task async def task_coro(value): # sleep for a moment await asyncio.sleep(2) # report a message print(f'>{value} done') # return the result return value * 100 # coroutine used for the entry point async def main(): # report a message print('Main starting') # create many coroutines coros = [task_coro(i) for i in range(10)] # run the tasks in the background results = await asyncio.gather(*coros) # report results print(results) # report a message print('Main done') # start the asyncio program asyncio.run(main()) |
Running the example first starts the asyncio event loop and runs the main() coroutine.
The main() coroutine runs and reports a message. It then creates a list of 10 coroutines with different arguments and executes them concurrency with asyncio.gather(). The main() coroutine suspends until all coroutines are done.
Each custom coroutine task runs, suspending for a moment, reporting a message, and then returning a result.
All custom coroutines are complete and the main() coroutine resumes. The list of return values is printed and a final message is reported.
This highlights the typical usage of asyncio.gather(), executed in the foreground, suspending until all issued coroutines are done.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 |
Main starting >0 done >1 done >2 done >3 done >4 done >5 done >6 done >7 done >8 done >9 done [0, 100, 200, 300, 400, 500, 600, 700, 800, 900] Main done |
Next, let’s look at how we might execute asyncio.gather() in the background.
Free Python Asyncio Course
Download your FREE Asyncio PDF cheat sheet and get BONUS access to my free 7-day crash course on the Asyncio API.
Discover how to use the Python asyncio module including how to define, create, and run new coroutines and how to use non-blocking I/O.
Example of asyncio.gather() In The Background
We can explore how to execute asyncio.gather() in the background.
In this case, we can update the above example to no longer await asyncio.gather() directly. Instead, we can assign the returned asyncio.Future and proceed with other tasks. Later, once we know that all issued coroutines are done, we can report the return values.
|
1 2 3 4 5 6 7 |
... # get the future for the gather future = asyncio.gather(*coros) # await something else await asyncio.sleep(3) # report results print(future.result()) |
The asyncio.gather() is executed in the background and the main coroutine is free to proceed with other work, in this case, sleep for an extended period.
The updated main() coroutine with this change is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 |
# coroutine used for the entry point async def main(): # report a message print('Main starting') # create many coroutines coros = [task_coro(i) for i in range(10)] # get the future for the gather future = asyncio.gather(*coros) # await something else await asyncio.sleep(3) # report results print(future.result()) # report a message print('Main done') |
Tying this together the complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 |
# SuperFastPython.com # example of running gather running in the background import asyncio # coroutine used for a task async def task_coro(value): # sleep for a moment await asyncio.sleep(2) # report a message print(f'>{value} done') # return the result return value * 100 # coroutine used for the entry point async def main(): # report a message print('Main starting') # create many coroutines coros = [task_coro(i) for i in range(10)] # get the future for the gather future = asyncio.gather(*coros) # await something else await asyncio.sleep(3) # report results print(future.result()) # report a message print('Main done') # start the asyncio program asyncio.run(main()) |
Running the example first starts the asyncio event loop and runs the main() coroutine.
The main() coroutine runs and reports a message. It then creates a list of 10 coroutines with different arguments and executes them concurrency with asyncio.gather().
The asyncio.Future from asyncio.gather() is assigned and main() does not suspend. Instead, it proceeds with other work, in this case suspending with a sleep for 3 seconds.
Each custom coroutine task runs, suspending for a moment, reporting a message, and then returning a result.
All custom coroutines then complete.
Separately, the main() coroutine resumes. The list of return values is retrieved from the asyncio.Future and printed before a final message is reported.
This highlights how we can run asyncio.gather() in the background.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 |
Main starting >0 done >1 done >2 done >3 done >4 done >5 done >6 done >7 done >8 done >9 done [0, 100, 200, 300, 400, 500, 600, 700, 800, 900] Main done |
Next, let’s look at a common mistake made when attempting to run asyncio.gather() in the background.
Overwhelmed by the python concurrency APIs?
Find relief, download my FREE Python Concurrency Mind Maps
Example of asyncio.gather() In Background With asyncio.create_task()
In asyncio, a common way to run coroutines in the background is to schedule them as an independent task via asyncio.create_task().
For example:
|
1 2 3 |
... # schedule coroutine in the background task = asyncio.create_task(coro()) |
Therefore, we might assume that this is how we would run asyncio.gather() in the background.
For example:
|
1 2 3 4 5 6 7 |
... # get the future for the gather future = asyncio.gather(*coros) # schedule the future task = asyncio.create_task(future) # await something else await asyncio.sleep(3) |
This in fact is a mistake and results in an exception.
The reason is because asyncio.gather() is not a coroutine, it is a regular Python function that returns an awaitable, specifically an asyncio.Future.
The asyncio.create_task() only takes coroutines and will raise an exception if we provide it with a future.
The example below demonstrates this error.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 |
# SuperFastPython.com # example of running gather running in the background with create_task import asyncio # coroutine used for a task async def task_coro(value): # sleep for a moment await asyncio.sleep(2) # report a message print(f'>{value} done') # return the result return value * 100 # coroutine used for the entry point async def main(): # report a message print('Main starting') # create many coroutines coros = [task_coro(i) for i in range(10)] # get the future for the gather future = asyncio.gather(*coros) # schedule the future task = asyncio.create_task(future) # await something else await asyncio.sleep(3) # report results print(future.result()) # report a message print('Main done') # start the asyncio program asyncio.run(main()) |
Running the example first starts the asyncio event loop and runs the main() coroutine.
The main() coroutine runs and reports a message.
It then creates a list of 10 coroutines with different arguments.
The list is proved to asyncio.gather() and a future is returned.
This future is then provided to the asyncio.create_task() function.
This results in a TypeError exception and the program terminates.
The error occurs because asyncio.create_task() expects a coroutine object and receives an asyncio.Future object.
This highlights that we cannot use asyncio.create_task() to run asyncio.gather() in the background.
|
1 2 3 4 |
Main starting TypeError: a coroutine was expected, got <_GatheringFuture pending> _GatheringFuture exception was never retrieved future: <_GatheringFuture finished exception=CancelledError()> |
Takeaways
You now know how to run asyncio.gather() in the background.
Did I make a mistake? See a typo?
I’m a simple humble human. Correct me, please!
Do you have any additional tips?
I’d love to hear about them!
Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.
Photo by Fikri Rasyid on Unsplash
Do you have any questions?