Using a Threading Lock in Asyncio Results in a Deadlock

You cannot use threading.Lock mutex locks to protect critical sections in asyncio programs.

In this tutorial, you will discover what happens if we try to use threading mutex locks in asyncio programs in Python.

Let’s get started.

Table of Contents

Can We Use threading.Lock instead of asyncio.Lock

The threading.Lock provides a mutex that can be used to protect a critical section of code from concurrent execution by multiple threads.

You can learn more about the threading.Lock in the tutorial:

Threading Mutex Lock in Python

Similarly, the asyncio.Lock provides a mutex to protect critical sections from coroutines. The asyncio.Lock is not thread-safe, meaning it is only appropriate for use with coroutines.

class asyncio.Lock: Implements a mutex lock for asyncio tasks. Not thread-safe.
— Asyncio Synchronization Primitives

You can learn more about the asyncio.Lock in the tutorial:

Asyncio Mutex Lock

We may consider whether we can use a threading Lock in an asyncio program.

Can a threading Lock be used to protect a critical section from concurrent execution by multiple threads and multiple coroutines?

Let’s explore this question.

In the next section, we will try to protect a critical section in an asyncio program using a threading.Lock.

Run loops using all CPUs, download your FREE book to learn how.

Example of threading.Lock in Asyncio

We can explore what happens if we use a threading.Lock in an asyncio program.

We can define a task coroutine that is passed the lock. The lock is acquired to protect a critical section that reports a message and sleeps a moment.

The main coroutine creates the lock, creates a number of coroutines that will share and make use of the lock concurrently, and then executes all coroutines and waits for them to finish.

Tying this together, the complete example is listed below.

# SuperFastPython.com

# example of using a threading Lock in an asyncio program

from random import random

from threading import Lock

import asyncio

# task coroutine with a critical section

async def task(lock, num, value):

# acquire the lock to protect the critical section

with lock:

# report a message

print(f'>coroutine {num} got the lock, sleeping for {value}')

# block for a moment

await asyncio.sleep(value)

# entry point

async def main():

# create a shared lock

lock = Lock()

# create many concurrent coroutines

coros = [task(lock, i, random()) for i in range(10)]

# execute and wait for tasks to complete

await asyncio.gather(*coros)

# run the asyncio program

asyncio.run(main())

Running the example first creates the main() coroutines and executes it as the entry point into the asyncio program.

The main() coroutine runs, first creating the threading mutex lock.

A list comprehension is then used to create 10 coroutines, each passing the lock, a unique integer, and a random number.

The main() coroutine is then suspended and awaits the execution of all coroutines.

The coroutines executed. The first acquires the lock, reports a message, then sleeps a moment.

This allows another coroutine to resume. It attempts to acquire the lock.

The program freezes with a deadlock.

It must be killed manually using CONTROL-C.

A sample of the program output is listed below, showing the message reported by the first coroutine only.

1	>coroutine 0 got the lock, sleeping for 0.10450054983534196

What happened?

A thread can only acquire a mutex lock once.

Attempting to acquire a lock that itself already holds, results in a deadlock.

It is irrelevant whether the thread attempted to acquire the lock again in a coroutine or not.

We can make this clearer with a more focused example.

Download Now: Free Asyncio PDF Cheat Sheet

Example of A Thread Acquiring a Lock Twice

We can explore a simpler example of a thread attempting to acquire the same lock twice.

In this example, we define a function to execute in a new thread.

The function takes the lock as an argument, reports a message, acquires the lock, reports a message again, attempts to acquire it again, then reports a final message.

The complete example is listed below.

# SuperFastPython.com

# example of a thread attempting to acquire the same lock more than once

from threading import Thread

from threading import Lock

# function executed in a new thread

def task(lock):

# report a message

print('Acquiring lock')

# acquire the lock

with lock:

# report a message

print('Acquiring lock again')

# acquire the lock

with lock:

# report message

print('All done')

# create the lock

lock = Lock()

# create the thread

thread = Thread(target=task, args=(lock,))

# start the thread

thread.start()

# wait for the thread to finish

thread.join()

Running the example first creates the shared lock.

A new thread is created and configured to execute the task() function and is passed the shared lock as an argument. The thread is started and the main thread blocks until the new thread is terminated.

The new thread runs.

A message is reported and the lock is acquired.

A second message is reported and the thread attempts to acquire the lock again.

This fails. The thread waits forever for the lock to become available. It does not become available because it already holds the lock.

The program must be terminated manually, e.g. via CONTROL-C.

1 2	Acquiring lock Acquiring lock again

The threading.Lock is not reentrant, meaning a thread may not acquire the same lock again.

Doing so results in a deadlock.

You can learn more about this in the tutorial:

How to Identify a Deadlock in Python

This is what happens in our asyncio program in the previous section.

The thread attempts to acquire the same lock twice, resulting in a deadlock.

This highlights why we cannot use a threading.Lock in an asyncio program and instead must use an asyncio.Lock.

Next, let’s consider using a reentrant threading Lock.

Free Python Asyncio Course

Download your FREE Asyncio PDF cheat sheet and get BONUS access to my free 7-day crash course on the Asyncio API.

Discover how to use the Python asyncio module including how to define, create, and run new coroutines and how to use non-blocking I/O.

Learn more

Example of threading Reentrant Lock in Asyncio

A threading.Lock is not reentrant, meaning a thread cannot acquire the lock, then acquire it again.

Nevertheless, the threading API provides a reentrant lock via the threading.RLock class.

You can learn more about the threading.RLock class in the tutorial:

Threading RLock in Python

This lock can be acquired by a thread, then acquired again.

Can we use this to protect critical sections in our coroutines?

The example below updates the asyncio program to use the reentrant lock instead of the non-reentrant lock.

# SuperFastPython.com

# example of using a threading RLock in an asyncio program

from random import random

from threading import RLock

import asyncio

# task coroutine with a critical section

async def task(lock, num, value):

# acquire the lock to protect the critical section

with lock:

# report a message

print(f'>coroutine {num} got the lock, sleeping for {value}')

# block for a moment

await asyncio.sleep(value)

# entry point

async def main():

# create a shared lock

lock = RLock()

# create many concurrent coroutines

coros = [task(lock, i, random()) for i in range(10)]

# execute and wait for tasks to complete

await asyncio.gather(*coros)

# run the asyncio program

asyncio.run(main())

Running the example first creates the main() coroutines and executes it as the entry point into the asyncio program.

The main() coroutine runs, first creating the threading reentrant mutex lock.

A list comprehension is then used to create 10 coroutines, each passing the lock, a unique integer and a random number.

The main() coroutine is then suspended and awaits the execution of all coroutines.

The coroutines executed. The first acquires the lock, reports a message, then sleeps a moment.

This allows another coroutine to resume. It attempts to acquire the lock and does. It reports a message, then sleeps for a moment.

The next coroutine runs, and so on.

All coroutines execute as fast as they are able and the lock is acquired immediately each time.

>coroutine 0 got the lock, sleeping for 0.4568901431793604

>coroutine 1 got the lock, sleeping for 0.29864186932986536

>coroutine 2 got the lock, sleeping for 0.09374909488228744

>coroutine 3 got the lock, sleeping for 0.4073937933735554

>coroutine 4 got the lock, sleeping for 0.5205997886127717

>coroutine 5 got the lock, sleeping for 0.38918445371545385

>coroutine 6 got the lock, sleeping for 0.17515599302225426

>coroutine 7 got the lock, sleeping for 0.4054819258792076

>coroutine 8 got the lock, sleeping for 0.36510786246727756

>coroutine 9 got the lock, sleeping for 0.13441856138988206

What happened?

The lock is thread-reentrant.

Asyncio uses a single thread to execute all coroutines.

Therefore, the same thread attempts to acquire the reentrant lock each time a coroutine reaches the critical section. This is allowed, the thread reenters the lock, and the critical section is not protected from concurrent execution.

This highlights that we cannot use threading mutex locks, reentrant, or otherwise to protect critical sections from concurrent execution by coroutines.

Overwhelmed by the python concurrency APIs?
Find relief, download my FREE Python Concurrency Mind Maps

Mixing Threads and Coroutines

We now know that we cannot use threading concurrency primitives to protect coroutines from race conditions.

This supports the need for asyncio-specific mutex locks.

What if we wanted to mix coroutines and threads?

For example, we may have a resource used by multiple coroutines across multiple threads.

That is, one asyncio event loop per thread, and multiple coroutines per event loop, all accessing the same critical section or resource.

I would strongly recommend separating the concerns of coroutines and threads.

For example, perhaps access to the resource can be limited to threads, such as in a thread pool, protected by a thread-safe mutex and coordinated by coroutines.

Or the verse, accessed only via coroutines and coordinated by threads, such as via a message queue.

If you are exploring this use case, let me know and I will develop some further examples.