You can find all stuck long-running tasks in asyncio by manually tracking how long each task has been alive and reporting task details if a threshold “too long” time is exceeded.

This approach can be used to find all stuck, hanging, and zombie asyncio tasks in the event loop, as well as those tasks that are blocked, but not yet not done.

In this tutorial, you will discover how to detect and report stuck long-running asyncio tasks.

Let’s get started.

Need to Find All Stuck and Long-Running Asyncio Tasks

Tasks can take too long in asyncio.

A coroutine may get stuck, waiting forever for some condition that will never occur.

• Maybe a resource is broken, or a connection is hanging.
• Maybe there is a race condition.
• Maybe there is a bug in a third-party library.

The Python asyncio module does provide some visibility of tasks that take too long.

This can be achieved by running the event loop in debug mode.

The problem is:

1. Running the event loop in debug mode is slower, a lot slower on large systems with many tasks.
2. Long-running tasks are only logged after they complete.
3. “Long-running” means blocking the event loop, not a zombie still hanging around in the event loop.

You can learn more about running the asyncio event loop in debug mode in the tutorial:

• How to Debug Asyncio

Therefore, we need ways of introspecting asyncio tasks that have been around for too long but have not stopped or been canceled.

The API does not provide this ability (and I cannot find a good third-party library that does this, yet), therefore we must develop something ourselves.

How to Discover Long-Running Tasks

We can explore a series of examples to approach the problem of finding and reporting asyncio tasks that have been running for too long.

Our approaches fall short of digging into the internals of event loops, we will stick with the simple task introspection tools provided by the high-level asyncio API.

We can stagger this effort into three levels, they are:

1. Report the details for all running tasks (e.g. names, stack traces, etc.)
2. Explicitly record the start time for each task and monitor tasks that have been running too long.
3. Monitor all active tasks and report on those running too long.

Let’s take a closer look at each step in turn.

Report Details of All Running Tasks

The first step is to introspect all running tasks.

We can get a list of all tasks currently running in the event loop by calling asyncio.all_tasks() and enumerating the result.

For example:

You can learn more about this function in the tutorial:

• How to Get All Asyncio Tasks in Python

For each task, we can report some information.

We could print or log each task, which gives some useful information, such as:

1. The task name
2. The coroutine used to create it, the script, and the line number
3. The status of the task
4. …and more.

For example:

Might give something like:

Another approach is to report the full stack trace of each task.

This can give an idea of where each task is currently at, such as the line on which it is stuck.

We can achieve this via the Task.print_stack() method.

For example:

We can also send the trace to the log by first capturing it in an io.StringIO buffer, then writing the buffer to the logging infrastructure.

For example:

We can tie all of this together into a helper function that can exclude some tasks from the list (like the main task) and can be called whenever we want to dump the details of all tasks on demand, such as periodically every few minutes.

Next, let’s look at monitoring how long all tasks have been running and only reporting the details that have been running too long.

Record When All Tasks Are Started And Monitor

Another approach is to explicitly record the start time for each task executed in the asyncio program.

This is a big task but might be possible for small and moderately sized programs.

It requires defining a helper function used to create and issue all tasks. It uses a dictionary global variable to store the task name and start time (using some reliable clock like the event loop time or similar).

For example

This assumes that each task has a unique name, which is probably true if you’re using default task names.

The TASK_DICT global variable must be declared and defined somewhere early in your program.

We can then define a monitor coroutine.

This coroutine runs forever in a loop. Each iteration it checks all coroutines that are running checks how long they have been running, then logs the details of those that have been alive for too long.

Any task that is running and is not known to the TASK_DICT dict, is ignored. We can also have an exclusion list for tasks we know run a long time, like the main() coroutine.

In this case, we report the task name.

This could just as easily be the string representation of the entire task or the entire stack trace for the task, as in the previous section.

The monitor loop also runs every second, but this could be every minute or every 10 minutes for long-running asyncio programs.

We can then kick off this monitor coroutine at the beginning of our program.

We must also explicitly cancel it when shutting down our program.

The obvious limitation of this approach is the need to centralize the creation of all asyncio tasks via the helper function.

Next, let’s look at removing this limitation.

Monitor All Active Tasks

Extending the above, the monitor alone can keep track of all active tasks over time.

This removes the need for explicitly recording the start time of tasks via the helper function.

Instead, the monitor records the first time a task is seen, then reports task details for those tasks that have been running too long.

For example, the main loop of the monitor() coroutine is updated to add the task to the dict if not seen before, rather than skipping over it.

The updated monitor with this change is listed below.

As above, the monitor can be started when the asyncio program is first started, excluding the main coroutine. Then canceled right before exiting.

You could put this in a context manager or a simple try-finally block.

Now that we know how to find stuck and zombie asyncio tasks that have been alive for a long time, let’s look at some worked examples.

Example of Logging a Trace For All Current Tasks

We can explore an example that reports a trace for all currently active asyncio tasks.

In this example, we will define a simple coroutine worker task that takes an argument, sleeps for 5 seconds, then returns a result.

We can then create a TaskGroup and issue 3 of these tasks for execution and give them a few seconds to execute.

If you are new to asyncio.TaskGroup, see the tutorial:

• How to use asyncio.TaskGroup

Then, while the tasks are still running, we can call our log_all_tasks_helper() function defined above. The current task main() will be excluded from the report.

This will report stack information for all 3 running work() coroutines.

The log_all_tasks_helper() is a function that does some blocking IO, e.g. printing to standard out. This could be performed in a new thread (e.g. to_thread()) if it is expected to spend a long time printing.

We can then configure simple logging and start the event loop.

Tying this together, the complete example is listed below.

Running the example first issues the three work() coroutines for execution and then suspends for 3 seconds.

The work() coroutines begin executing and suspend for 5 seconds.

The main() coroutine resumes and then calls our custom log_all_tasks_helper() function.

This reports stack information for all 3 currently running tasks and the line on which they are currently suspended.

This highlights how we can introspect the state of all running tasks in our asyncio programs.

Next, let’s explore an example of explicitly recording the start time of all tasks.

Example of Recording Start Time for All Tasks

We can explore an example of explicitly recording the start time of all asyncio tasks in a program.

This may be required in cases where we cannot trust the monitor to determine how long tasks have been running (in the next section), and the program is small enough that we can centralize the creation and scheduling of all tasks in the program via our start_task() helper function, developed above.

The monitor() coroutine, also developed above, will run for the life of the program and report the details of those tasks that have been alive for “too long”, in this case, 3 seconds or more.

We will just report the task name and time alive, but this could be updated to report all task details or even the task stack trace.

Firstly, we will define a do_work() coroutine that will sleep for a random number of seconds between 1 and 10.

This will give some variety in results each time we run the example. Some tasks will take a long time, some won’t.

Next, in the main() coroutine, we will create and start the monitor coroutine, then issue 10 of our do_work() coroutines.

Importantly, the work tasks are created using our custom start_task() helper function.

The main() coroutine will then suspend and wait for all tasks to complete.

Once all tasks are done, the monitor will be explicitly canceled.

Finally, logging is enabled and the event loop is started.

The monitor() coroutines require a global variable to keep track of task start times. This can be declared and defined before starting the event loop.

Tying this together, the complete example is listed below.

Running the example first starts the monitor() coroutine, configured to report any tasks that take longer than 3 seconds, and ignores the main() coroutine.

Next, all 3 worker tasks are created and scheduled using our helper function. This records the start time of each task.

The main() coroutine then suspends waiting for the tasks to complete.

The tasks execute, each sleeping for a random number of seconds.

The monitor() coroutine runs every second and checks how long each task has been alive. As soon as some tasks have been seen to be alive for “too long”, debug log messages are reported.

This continues until all tasks are completed.

We can see by the end only a handful of tasks have been alive for a long time, e.g. 10 seconds.

Before exiting, the main() coroutine cancels the monitor task.

This highlights how we can explicitly record task start times, then monitor all known tasks and report debug messages when they have been alive for too long.

This approach can be used to aromatically discover long-running tasks in an asyncio program.

Next, let’s look at an example that does not require explicit recording of start times via a helper function.

Example of Monitoring All Active Tasks

We can explore an example of automatically monitoring all active tasks in an asyncio program.

This can be achieved by keeping track of tasks the first time they are seen, then reporting on those tasks if they have been around for too long, e.g. are still active after a given time limit.

We will use the updated monitor() coroutine developed above.

In this case, we do not need to define a global variable dict for tracking tasks and create times, this state is kept within the monitor() coroutine.

We will use the same work tasks from the previous section that run for a random number of seconds.

Similarly, the monitor() coroutine can be created at the beginning of the program and canceled before shutdown.

Tying this together, the complete example is listed below.

Running the example first starts the monitor() coroutine, configured to report any tasks that take longer than 3 seconds, and ignore the main() coroutine.

Next, all 3 worker tasks are created and scheduled normally, not using any helper function.

The main() coroutine then suspends waiting for the tasks to complete.

The tasks execute, each sleeping for a random number of seconds.

The monitor() coroutine runs every second. The first time each task is seen, it is recorded in the dict, including its name and timestamp.

The monitor then checks how long each task has been alive. As soon as some tasks have been seen to be alive for “too long”, debug log messages are reported.

This continues until all tasks are completed.

We can see by the end only a handful of tasks have been alive for a long time, e.g. 9 and 10 seconds.

Before exiting, the main() coroutine cancels the monitor task.

This example highlights how we can report the details of any task that has been alive for “too long” in an asyncio program by adding a simple monitoring coroutine.

Further Reading

This section provides additional resources that you may find helpful.

Python Asyncio Books

• Python Asyncio Mastery, Jason Brownlee (my book!)
• Python Asyncio Jump-Start, Jason Brownlee.
• Python Asyncio Interview Questions, Jason Brownlee.
• Asyncio Module API Cheat Sheet

I also recommend the following books:

• Python Concurrency with asyncio, Matthew Fowler, 2022.
• Using Asyncio in Python, Caleb Hattingh, 2020.
• asyncio Recipes, Mohamed Mustapha Tahrioui, 2019.

Guides

• Python Asyncio: The Complete Guide
• Python Asynchronous Programming

APIs

• asyncio — Asynchronous I/O
• Asyncio Coroutines and Tasks
• Asyncio Streams
• Asyncio Subprocesses
• Asyncio Queues
• Asyncio Synchronization Primitives

References

• Asynchronous I/O, Wikipedia.
• Coroutine, Wikipedia.

Takeaways

You now know how to detect and report on stuck long-running asyncio tasks.

Did I make a mistake? See a typo?
I’m a simple humble human. Correct me, please!

Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.

Photo by Varla Scooter on Unsplash