We can run an asyncio event loop in a new thread by starting a new thread and configuring it to start or run an event loop.
There are many approaches we can use to run an event loop in a new thread. The simplest is to configure a new thread to start an event loop and run a given coroutine.
Another approach is to create a new event loop and provide it to a separate daemon thread to run in the background. We can then issue coroutines to the event loop in a separate thread for ad hoc execution.
In this tutorial, you will discover how to run an asyncio event loop in a separate thread.
Let’s get started.
Asyncio Event Loop in Separate Thread
Typically the asyncio event loop runs in the main thread.
Recall that the main thread is the thread used to start and run the Python interpreter.
You can learn more about the main thread in the tutorial:
We can also run the asyncio event loop in a new thread separate from the main thread.
Running an asyncio event loop in a separate thread can be helpful in certain scenarios, particularly when we want to integrate asyncio-based asynchronous code with synchronous code or libraries that are not asyncio-compatible.
Some reasons why we might choose to run an asyncio event loop in a separate thread include:
- Integration with synchronous code: If we have existing synchronous code that we want to integrate with asyncio, running the event loop in a separate thread allows us to do so without blocking the main thread. This is especially useful in applications where we need to handle both synchronous and asynchronous tasks concurrently.
- Interoperability with blocking I/O operations: Many libraries and APIs are built around synchronous blocking I/O operations. Running the asyncio event loop in a separate thread enables us to use asyncio for non-blocking I/O operations while still being able to interact with these blocking I/O operations in the main thread without causing blocking issues.
- Isolation: Running the asyncio event loop in a separate thread provides a level of isolation, making it easier to reason about and manage asynchronous code separately from the main application logic. This can lead to cleaner and more maintainable code, especially in complex applications.
Running the asyncio event loop in a separate thread does not come without a cost. It adds additional complexity to the program and can introduce potential pitfalls such as race conditions, deadlocks, and synchronization issues.
Careful consideration should be given to thread safety and proper synchronization mechanisms when working with asyncio in a multi-threaded environment.
Next, let’s look at how we might run the asyncio event loop in a separate thread.
Run loops using all CPUs, download your FREE book to learn how.
How to Run The Event Loop in a Separate Thread
There are several ways to run an asyncio event loop in a new thread.
Some of the more common approaches include the following:
- Call asyncio.run() Directly
- Call A Helper Function
- Background Event Loop And Send Coroutines
Let’s take a closer look at each approach in turn.
Call asyncio.run() Directly
A simple approach is to start a new thread and configure the thread to start a new asyncio event loop and pass it a coroutine to execute.
This can be achieved by creating a new threading.Thread instance. The “target” argument can be set to run the asyncio.run module function and a coroutine object can be provided via the “args” argument.
Once created, the new thread can be started via the start() method.
For example:
|
1 2 3 4 5 |
... # create a new thread to execute a target coroutine thread = threading.Thread(target=asyncio.run, args=(main(),)) # start the new thread thread.start() |
The benefit of this approach is that it is very simple and does not require additional code. We can specify the coroutine we wish to execute directly when configuring the new thread.
You can learn more about starting a new thread in the tutorial:
You can learn more about running an asyncio program in the tutorial:
Call A Helper Function
Another approach is to define a new helper function that will in turn create and start a new asyncio event loop.
For example:
|
1 2 3 4 |
# helper function to start and run the asyncio event loop def run_event_loop(): # start the event loop asyncio.run(main_coroutine()) |
This helper function can then be used as the target function when creating and starting a new thread.
For example:
|
1 2 3 4 5 |
... # create a new thread to execute a target coroutine thread = threading.Thread(target=run_event_loop) # start the new thread thread.start() |
The benefit of this approach is that it gives a little more control over the setup and start of the new asyncio event loop.
For example, if the event loop requires additional configuration, such as debugging, logging, or similar then this can be configured before starting the event loop.
It also provides a point of change if an alternate approach is preferred for starting the event loop, such as using the low-level asyncio API.
Background Event Loop And Send Coroutines
Another approach is to first create a new event loop, then pass the event loop to a separate thread to run forever.
This requires defining a helper function that takes the new event loop object and is responsible for blocking and running the event loop forever via the loop.run_forever() method.
You can learn more about running forever in the tutorial:
It is also a good practice for the thread to set the event loop as the default loop for the thread, in case any coroutines perform introspection. This can be achieved by calling the asyncio.set_event_loop() function prior to running forever.
|
1 2 3 4 5 6 |
# helper function to start and run the asyncio event loop def run_event_loop(loop): # set the loop for the current thread asyncio.set_event_loop(loop) # run the event loop until stopped loop.run_forever() |
We can then create a new event loop using the low-level API via the asyncio.new_event_loop() function.
|
1 2 3 |
... # create a new event loop (low-level api) loop = asyncio.new_event_loop() |
Next, we can create a new thread and configure it to run our run_event_loop() helper function and pass it the new event loop object.
We can also configure the new thread to be a daemon thread so that if we close the main program while the event loop is running, the event loop will be terminated.
|
1 2 3 4 5 |
... # create a new thread to execute a target coroutine thread = threading.Thread(target=run_event_loop, args=(loop,), daemon=True) # start the new thread thread.start() |
You can learn more about daemon threads in the tutorial:
So far, the event loop is created and running in a separate thread but is not executing anything.
We can create coroutines and send them to the event loop for execution via the asyncio.run_coroutine_threadsafe() method.
For example:
|
1 2 3 4 |
... future = asyncio.run_coroutine_threadsafe(task(), loop) # wait for the task to finish value = future.result() |
You can learn more about running coroutines from a separate thread in the tutorial:
The benefit of this approach is that we can have a helper asyncio event loop running in a separate thread and have it execute ad hoc coroutines on demand.
This is helpful if we need to run occasional asyncio code within a non-asyncio program.
Now that we know how to run an asyncio event loop in a separate thread, let’s look at some examples.
Example of Asyncio Event Loop in a Separate Thread Directly
We can explore an example of running an asyncio event loop in a separate thread directly.
In this example, we can first define a custom coroutine that loos 5 times, reports a message and suspends for one second for each iteration.
|
1 2 3 4 5 6 7 8 |
# main coroutine for the asyncio program async def main_coroutine(): # run the event loop a whole for _ in range(5): # report a message print('>>Asyncio is running') # suspend a moment await asyncio.sleep(1) |
We can then create a new thread and configure it to start a new asyncio event loop and execute our custom coroutine.
The new thread can then be started.
|
1 2 3 4 5 |
... # create a new thread to execute a target coroutine thread = threading.Thread(target=asyncio.run, args=(main_coroutine(),)) # start the new thread thread.start() |
Finally, the main thread can then continue on with other tasks, in this case, it will loop 10 times, report a message, and sleep for a fraction of a section each iteration.
|
1 2 3 4 5 6 7 |
... # run other tasks for some time for _ in range(10): # report a message print('>Main thread is running') # sleep a moment time.sleep(0.5) |
The print messages from both threads will show that the main thread and the asyncio event loop in the separate thread are running concurrently.
Tying this together, the complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 |
# SuperFastPython.com # example of running the asyncio event loop in a separate thread import threading import asyncio import time # main coroutine for the asyncio program async def main_coroutine(): # run the event loop a whole for _ in range(5): # report a message print('>>Asyncio is running') # suspend a moment await asyncio.sleep(1) # create a new thread to execute a target coroutine thread = threading.Thread(target=asyncio.run, args=(main_coroutine(),)) # start the new thread thread.start() # run other tasks for some time for _ in range(10): # report a message print('>Main thread is running') # sleep a moment time.sleep(0.5) |
Running the example first creates and configures a new thread to start an event loop and run our custom coroutine.
The new thread is then started.
The main thread then continues on and begins looping, reporting messages, and sleeping.
The new thread starts in the background. It creates and starts a new asyncio event loop and runs the custom coroutine. The coroutine runs its loop, reporting messages and suspending each iteration.
Each loop runs until completion and once the main thread and the new thread are done, the program terminates.
This highlights how we can run ad hoc coroutines in an asyncio event loop in a separate thread from the main thread.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 |
>Main thread is running >>Asyncio is running >Main thread is running >>Asyncio is running >Main thread is running >Main thread is running >>Asyncio is running >Main thread is running >Main thread is running >>Asyncio is running >Main thread is running >Main thread is running >>Asyncio is running >Main thread is running >Main thread is running |
Next, let’s look at an example of how we might achieve the same result using a helper function.
Free Python Asyncio Course
Download your FREE Asyncio PDF cheat sheet and get BONUS access to my free 7-day crash course on the Asyncio API.
Discover how to use the Python asyncio module including how to define, create, and run new coroutines and how to use non-blocking I/O.
Example of Event Loop in a Thread With Helper Function
We can explore an example of running an asyncio event loop in a separate thread using a helper function.
In this case, we can update the previous example to define a helper function that will start the asyncio event loop and run our custom coroutine.
Firstly, we can define the helper function.
|
1 2 3 4 |
# helper function to start and run the asyncio event loop def run_event_loop(): # start the event loop asyncio.run(main_coroutine()) |
We can then update the configuration of the new thread to call our helper function in the new thread.
|
1 2 3 |
... # create a new thread to execute a target coroutine thread = threading.Thread(target=run_event_loop) |
Tying this together, the complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 |
# SuperFastPython.com # example of running the asyncio event loop in a separate thread import threading import asyncio import time # main coroutine for the asyncio program async def main_coroutine(): # run the event loop a whole for _ in range(5): # report a message print('>>Asyncio is running') # suspend a moment await asyncio.sleep(1) # helper function to start and run the asyncio event loop def run_event_loop(): # start the event loop asyncio.run(main_coroutine()) # create a new thread to execute a target coroutine thread = threading.Thread(target=run_event_loop) # start the new thread thread.start() # run other tasks for some time for _ in range(10): # report a message print('>Main thread is running') # sleep a moment time.sleep(0.5) |
Running the example first creates and configures a new thread to run our helper function.
The new thread is then started.
The main thread then continues on and begins looping, reporting messages, and sleeping.
The new thread starts in the background. The helper function is called and it creates and starts a new asyncio event loop and runs the custom coroutine.
The coroutine runs its loop, reporting messages and suspending each iteration.
Each loop runs until completion and once the main thread and the new thread are done, the program terminates.
This highlights how we can use a helper function to run coroutines in an asyncio event loop in a separate thread from the main thread.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 |
>Main thread is running >>Asyncio is running >Main thread is running >Main thread is running >>Asyncio is running >Main thread is running >Main thread is running >>Asyncio is running >Main thread is running >Main thread is running >>Asyncio is running >Main thread is running >>Asyncio is running >Main thread is running >Main thread is running |
Next, let’s look at an example of running a background event loop that we can interact with directly.
Overwhelmed by the python concurrency APIs?
Find relief, download my FREE Python Concurrency Mind Maps
Example of Running Coroutine in Background Event Loop
We can example of creating an event loop and then running it in the background and sending it coroutines for ad hoc execution.
In this case, we can define a new task that reports a message, suspends a moment, and returns a value.
|
1 2 3 4 5 6 7 8 |
# coroutine to simulate some work async def task(): # report a message print('Task is running in the event loop') # suspend a moment await asyncio.sleep(1) # return a value return 100 |
We can then define a helper function to run the asyncio event loop.
The helper function takes a loop as an argument. It reports a message, configures the thread to use the provided loop as the default loop for the thread, then runs the event loop forever, waiting for tasks to execute.
|
1 2 3 4 5 6 7 8 |
# helper function to start and run the asyncio event loop def run_event_loop(loop): # report a message print('Asyncio event loop is running') # set the loop for the current thread asyncio.set_event_loop(loop) # run the event loop until stopped loop.run_forever() |
Next, we can create a new event loop.
|
1 2 3 |
... # create a new event loop (low-level api) loop = asyncio.new_event_loop() |
Finally, we can create a new thread and configure it to run our helper function and pass it the event loop object. We will also configure the thread to be a daemon thread so that it will not prevent the main thread from exiting when it is done.
|
1 2 3 |
... # create a new thread to execute a target coroutine thread = threading.Thread(target=run_event_loop, args=(loop,), daemon=True) |
Tying this together, the complete example is listed below.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 |
# SuperFastPython.com # example of running the asyncio event loop in a separate thread import threading import asyncio import time # coroutine to simulate some work async def task(): # report a message print('Task is running in the event loop') # suspend a moment await asyncio.sleep(1) # return a value return 100 # helper function to start and run the asyncio event loop def run_event_loop(loop): # report a message print('Asyncio event loop is running') # set the loop for the current thread asyncio.set_event_loop(loop) # run the event loop until stopped loop.run_forever() # create a new event loop (low-level api) loop = asyncio.new_event_loop() # create a new thread to execute a target coroutine thread = threading.Thread(target=run_event_loop, args=(loop,), daemon=True) # start the new thread thread.start() # run an asyncio task in the background event loop future = asyncio.run_coroutine_threadsafe(task(), loop) # wait for the task to finish value = future.result() # report a message print(f'Got Async Result: {value}') |
Running the example first creates a new event loop.
Next, a new thread is created and configured to run our helper function, pass the loop object as an argument, and for the thread to run as a background or daemon thread. The new thread is started.
The new thread starts and runs our helper function and reports a message. It then sets the default event loop for the thread to the provided loop and suspends, running the event loop forever.
The main thread then issues our task() coroutine to the event loop and waits for it to complete.
The coroutine is sent to the event loop running in a separate thread and executes. It reports a message, suspends for one second, then returns a value.
Interestingly, the call to the result() method on the future does not raise an InvalidStateError exception because the task is not yet done. This is because the Future is not an asyncio.Future, but instead a concurrent.futures.Future which will suspend the caller until the result is available.
You can learn more about the difference between the two Future objects in the tutorial:
The result is reported and the main thread exits. This terminates the daemon thread running the event loop in the background and the program exits.
This highlights how we can run an event loop in the background and issue ad hoc coroutines to it for execution.
|
1 2 3 |
Asyncio event loop is running Task is running in the event loop Got Async Result: 100 |
Takeaways
You now know how to run an asyncio event loop in a separate thread.
Did I make a mistake? See a typo?
I’m a simple humble human. Correct me, please!
Do you have any additional tips?
I’d love to hear about them!
Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.
Photo by Josh Sorenson on Unsplash
Do you have any questions?