Last Updated on November 14, 2023
It is common to issue many tasks at once, then need to process the results from each task as the tasks are completed.
This can be more efficient than waiting for all tasks to complete before handling the results.
We can achieve this and iterate tasks in the order they are done using the as_completed() function.
In this tutorial, you will discover how to use the as_completed() function in asyncio.
After completing this tutorial, you will know:
- How to use the as_completed() to iterate over a collection of tasks in the order they are completed.
- How to retrieve results from coroutines and tasks that are traversed in completion order.
- How to set and use a timeout when iterating tasks in completion order.
Let’s get started.
What is Asyncio as_completed()
The asyncio.as_completed() function will run a collection of tasks and coroutines concurrently.
More importantly, it returns an iterable that we can use to retrieve the awaitables in the order that they are completed.
Run awaitable objects in the aws iterable concurrently. Return an iterator of coroutines. Each coroutine returned can be awaited to get the earliest next result from the iterable of the remaining awaitables.
— Coroutines and Tasks
It allows us to execute many coroutines or tasks concurrently and get the results from tasks in the order that they are completed, rather than the order we issued them.
Now that we know what as_completed() is, let’s look at how we might use it.
Run loops using all CPUs, download your FREE book to learn how.
How to Use Asyncio as_completed()
The asyncio.as_completed() function is called with a collection of awaitables.
This may be a list, dict, or set, and may contain asyncio.Task objects, coroutines, or other awaitables.
Any coroutines provided to as_completed() will be wrapped in a Task object for independent execution.
It returns an iterable that when traversed will yield awaitables (or wrapped awaitables) in the provided list. These can be awaited by the caller in order to get results in the order that tasks are completed, e.g. get the result from the next task to complete.
For example:
|
1 2 3 4 5 |
... # iterate over awaitables for task in asyncio.as_completed(tasks): # get the next result result = await task |
The as_completed() also takes a “timeout” argument.
This specifies how long the caller is willing to wait for all awaitables to be done.
For example:
|
1 2 3 4 5 |
... # iterate over awaitables with a timeout for task in asyncio.as_completed(tasks, timeout=10): # get the next result result = await task |
If the timeout elapses before all awaitables are done, a asyncio.TimeoutError is raised and may need to be handled.
For example, we could handle it within the loop:
|
1 2 3 4 5 6 7 8 9 |
... # iterate over awaitables with a timeout for task in asyncio.as_completed(tasks, timeout=10): # handle a timeout try: # get the next result result = await task except asyncio.TimeoutError: # ... |
This is not desirable because once the timeout has elapsed, an asyncio.TimeoutError will be raised each time next() is called on the generator.
Therefore, it is better to wrap the entire loop in a try-except block.
For example:
|
1 2 3 4 5 6 7 8 9 |
... # handle a timeout try: # iterate over awaitables with a timeout for task in asyncio.as_completed(tasks, timeout=10): # get the next result result = await task except asyncio.TimeoutError: # ... |
Now that we know how to use the as_completed() function, let’s take a moment to consider how it works.
How does Asyncio as_completed() Work
The works by providing a generator that yields coroutines, where each coroutine will return a result of a provided awaitable.
The asyncio.as_completed() function does not block (suspend the current coroutine) but instead returns a generator.
For example:
|
1 2 3 |
... # get a generator that yields awaitables in completion order generator = asyncio.as_completed(tasks) |
Calling the next() built-in function on the generator does not block, but instead yields a coroutine.
The returned coroutine is not one of the provided awaitables, but rather an internal coroutine from the as_completed() function that manages and monitors which issued task will return a result next.
For example:
|
1 2 3 |
... # get the next coroutine coro = next(generator) |
It is not until one of the returned coroutines is awaited that the caller will block.
For example:
|
1 2 3 |
... # get a result from the next task to complete result = await coro |
Now that we have an idea of how to use the as_completed() function and how it works, let’s look at some worked examples.
Free Python Asyncio Course
Download your FREE Asyncio PDF cheat sheet and get BONUS access to my free 7-day crash course on the Asyncio API.
Discover how to use the Python asyncio module including how to define, create, and run new coroutines and how to use non-blocking I/O.
Example of as_completed() with Coroutines
We can explore how to execute coroutines concurrently and get coroutine results as tasks are completed with the asyncio.as_completed() function.
In this example, we will define a simple coroutine task that takes an integer argument, generates a random value, sleeps for a fraction of a second then returns the integer argument multiplied by the generated value.
A list of the task coroutine is created and passed to the as_completed() function. This returns a generator that is traversed using a for loop.
In each iteration of the loop a coroutine is yielded from the generator and is then awaited in the body of the loop. A result from the next coroutine to complete is returned and the result is reported.
The complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 |
# SuperFastPython.com # example of getting coroutine results as completed from random import random import asyncio # coroutine to execute in a new task async def task_coro(arg): # generate a random value between 0 and 1 value = random() # block for a moment await asyncio.sleep(value) # return the result return arg * value # main coroutine async def main(): # create many coroutines coros = [task_coro(i) for i in range(10)] # get results as coroutines are completed for coro in asyncio.as_completed(coros): # get the result from the next coroutine to complete result = await coro # report the result print(f'>got {result}') # start the asyncio program asyncio.run(main()) |
Running the example first creates the main() coroutine and then uses this as the entry point into the asyncio program.
The main() coroutine runs and creates a list of coroutines.
It then passes this list to the as_completed() function which returns a generator. The generator is traversed in a for loop and each iteration yields a coroutine.
The coroutine is awaited. This suspends the main() coroutine.
The tasks begin executing, generating a random value, and sleeping. A task finishes and returns a value.
The main() coroutine resumes, receives the return value, and reports it.
The loop repeats, another coroutine is yielded, the main() coroutine awaits it and suspends, and another result is returned.
This continues until all coroutines in the provided list are completed.
This example highlights that we can traverse a collection of coroutines and get and use results in the order that tasks are completed.
Results will differ each time the example is run given the use of random numbers.
|
1 2 3 4 5 6 7 8 9 10 |
>got 0.07236962530232949 >got 0.5171864910147306 >got 0.7153626682953872 >got 2.54812333824902 >got 0.5960648114598495 >got 5.051883987489034 >got 0.0 >got 2.842043340472799 >got 6.343694133393031 >got 4.903128525746293 |
Next, let’s look at how we might use the as_completed() function with asyncio.Task objects.
Overwhelmed by the python concurrency APIs?
Find relief, download my FREE Python Concurrency Mind Maps
Example of as_completed() with Tasks
We can explore how to execute asyncio.Task objects concurrently and gets coroutine results as tasks are completed with the asyncio.as_completed() function.
In this example, we will update the above example to create a list of asyncio.Task objects instead of coroutines.
Recall that a Task object wraps a coroutine. Once created, a task is scheduled for execution immediately and will be executed independently.
You can learn more about asyncio tasks in the tutorial:
The complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 |
# SuperFastPython.com # example of getting task results as completed from random import random import asyncio # coroutine to execute in a new task async def task_coro(arg): # generate a random value between 0 and 1 value = random() # block for a moment await asyncio.sleep(value) # return the result return arg * value # main coroutine async def main(): # create many tasks coros = [asyncio.create_task(task_coro(i)) for i in range(10)] # get results as tasks are completed for coro in asyncio.as_completed(coros): # get the result from the next task to complete result = await coro # report the result print(f'>got {result}') # start the asyncio program asyncio.run(main()) |
Running the example first creates the main() coroutine and then uses this as the entry point into the asyncio program.
The main() coroutine runs and creates a list of Task objects, instead of coroutines.
Each task is scheduled for independent execution after it is created. This is different from the coroutines in the previous example that are not scheduled until the as_completed() function wraps them in Task objects.
The main() coroutine then passes this list to the as_completed() function which returns a generator. The generator is traversed in a for loop and each iteration yields a coroutine.
The coroutine is awaited. This suspends the main() coroutine.
The tasks begin executing, generating a random value, and sleeping. A task finishes and returns a value.
The main() coroutine resumes, receives the return value, and reports it.
This continues until all coroutines in the provided list are completed.
This example highlights that the as_completed() function can be used with Task objects as well as coroutines that we saw in the previous example.
Results will differ each time the example is run given the use of random numbers.
|
1 2 3 4 5 6 7 8 9 10 |
>got 0.3453975438487853 >got 0.5871779512599952 >got 0.0 >got 0.6767468346945187 >got 0.9905365580643459 >got 1.9274714575238143 >got 0.5127918804896211 >got 1.308261888746516 >got 4.12075681907987 >got 7.128076208913287 |
Example of as_completed() with a Timeout
We can explore how to execute coroutines concurrently with the as_completed() function and a timeout and handle cases where results take longer than the expected timeout.
In this example, we call the as_completed() function with a timeout that is large enough for all tasks to be completed.
This is intentional as we want to explore how to handle the case that a collection of tasks takes too long to be done.
The complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 |
# SuperFastPython.com # example of getting coroutine results as completed with a timeout from random import random import asyncio # coroutine to execute in a new task async def task_coro(arg): # generate a random value between 0 and 1 value = random() # block for a moment await asyncio.sleep(value) # return the result return arg * value # main coroutine async def main(): # create many coroutines coros = [task_coro(i) for i in range(10)] # handle a timeout try: # get results as coroutines are completed for coro in asyncio.as_completed(coros, timeout=0.5): # get the result from the next coroutine to complete result = await coro # report the result print(f'>got {result}') except asyncio.TimeoutError: print('Gave up after timeout') # start the asyncio program asyncio.run(main()) |
Running the example first creates the main() coroutine and then uses this as the entry point into the asyncio program.
The main() coroutine runs and creates a list of coroutines.
It then passes this list to the as_completed() function with a timeout.
A generator is returned and traversed in a for loop and each iteration yields a coroutine.
The coroutine is awaited. This suspends the main() coroutine.
The tasks begin executing, generating a random value, and sleeping. A task finishes and returns a value.
The main() coroutine resumes, receives the return value, and reports it.
This continues until the timeout elapses, after which, the next call to await a returned coroutine raises a TimeoutError exception.
This breaks the loop and is caught, reporting a message.
This example highlights how we can impose a limit on how long the caller is willing to wait for all tasks to be completed and handle the case where the timeout elapses before all results are retrieved.
Results will differ each time the example is run given the use of random numbers.
|
1 2 3 4 5 6 7 |
>got 1.1794285132416094 >got 0.2733126502808916 >got 0.9659219853827947 >got 1.8523793561409736 >got 0.0 >got 3.4477334386446854 Gave up after timeout |
Example of as_completed() with an Exception
We can explore how to execute coroutines concurrently with as_completed() and one of the tasks fails with an exception.
In this example, we update the above example so that one of the task coroutines intentionally fails with an exception.
Tasks can fail and it is important to be aware of this and to consider handling a task failure when waiting for results.
You can learn more about exceptions in asyncio tasks in the tutorial:
The complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 |
# SuperFastPython.com # example of getting coroutine results as completed with an exception from random import random import asyncio # coroutine to execute in a new task async def task_coro(arg): # generate a random value between 0 and 1 value = random() # block for a moment await asyncio.sleep(value) # check if the task should fail if value > 0.5: raise Exception('Something bad happened') # return the result return arg * value # main coroutine async def main(): # create many coroutines coros = [task_coro(i) for i in range(10)] # get results as coroutines are completed for coro in asyncio.as_completed(coros): # get the result from the next coroutine to complete result = await coro # report the result print(f'>got {result}') # start the asyncio program asyncio.run(main()) |
Running the example first creates the main() coroutine and then uses this as the entry point into the asyncio program.
The main() coroutine runs and creates a list of coroutines.
It then passes this list to the as_completed() function which returns a generator. The generator is traversed in a for loop and each iteration yields a coroutine.
The coroutine is awaited. This suspends the main() coroutine.
The tasks begin executing, generating a random value, and sleeping. A task finishes and returns a value.
The main() coroutine resumes, receives the return value, and reports it.
This continues for a number of iterations.
Then, one of the tasks fails with an exception. When it is awaited, the exception is re-raised by the caller.
The exception is not expected and therefore it unwinds the asyncio program and terminates the main thread.
This example highlights what happens if a task in the collection fails with an exception. It reminds us to be careful when awaiting results from a coroutine and to potentially handle exceptions that could be raised.
Results will differ each time the example is run given the use of random numbers.
|
1 2 3 4 5 6 7 8 |
>got 0.05191041233869953 >got 0.3949931321512907 >got 0.1518973920395813 >got 2.125424835725659 >got 0.0 Traceback (most recent call last): ... Exception: Something bad happened |
Example of as_completed() with “async for”
We can explore a common error where the “async for” expression is used with the generator returned from the as_completed() function.
The “async for” expression is used for asynchronous for-loops, specifically with asynchronous iterators and asynchronous generators.
You can learn more about the “async for” expression in the tutorial:
The as_completed() function does not return an asynchronous generator, it returns a traditional generator.
Therefore, using “async for” with the as_completed() function is an error.
In this example, we intentionally use the “async for” expression with the as_completed() function to demonstrate this error.
The complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 |
# SuperFastPython.com # example of getting coroutine results as completed and async for from random import random import asyncio # coroutine to execute in a new task async def task_coro(arg): # generate a random value between 0 and 1 value = random() # block for a moment await asyncio.sleep(value) # return the result return arg * value # main coroutine async def main(): # create many coroutines coros = [task_coro(i) for i in range(10)] # get results as coroutines are completed async for coro in asyncio.as_completed(coros): # get the result from the next coroutine to complete result = await coro # report the result print(f'>got {result}') # start the asyncio program asyncio.run(main()) |
Running the example first creates the main() coroutine and then uses this as the entry point into the asyncio program.
The main() coroutine runs and creates a list of coroutines.
It then passes them to the as_completed() function. This returns a generator that is iterated using an “async for” expression.
This fails because the as_completed() function returns a traditional generator, not an asynchronous generator.
The fix involves changing the “async for” expression to a traditional “for” expression.
For example:
|
1 2 3 4 5 |
... # get results as coroutines are completed for coro in asyncio.as_completed(coros): # get the result from the next coroutine to complete result = await coro |
This example highlights a common error when using the as_completd() function and how to overcome it.
|
1 2 3 4 |
Traceback (most recent call last): ... TypeError: 'async for' requires an object with __aiter__ method, got generator sys:1: RuntimeWarning: coroutine 'task_coro' was never awaited |
Further Reading
This section provides additional resources that you may find helpful.
Python Asyncio Books
- Python Asyncio Mastery, Jason Brownlee (my book!)
- Python Asyncio Jump-Start, Jason Brownlee.
- Python Asyncio Interview Questions, Jason Brownlee.
- Asyncio Module API Cheat Sheet
I also recommend the following books:
- Python Concurrency with asyncio, Matthew Fowler, 2022.
- Using Asyncio in Python, Caleb Hattingh, 2020.
- asyncio Recipes, Mohamed Mustapha Tahrioui, 2019.
Guides
APIs
- asyncio — Asynchronous I/O
- Asyncio Coroutines and Tasks
- Asyncio Streams
- Asyncio Subprocesses
- Asyncio Queues
- Asyncio Synchronization Primitives
References
Takeaways
You now know how to use the asyncio.as_completed() function in Python.
Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.
Photo by Marco Buschini on Unsplash
Do you have any questions?