Lock vs Semaphore in Python

Last Updated on September 12, 2022

Locks and Semaphores are two types of concurrency primitives.

In this tutorial you will discover the difference between the Lock and Semaphore and when to use each in your Python projects.

Let’s get started.

Table of Contents

What is Lock

A mutual exclusion lock or mutex lock is a synchronization primitive intended to prevent a race condition.

A race condition is a concurrency failure case when two threads run the same code and access or update the same resource (e.g. data variables, stream, etc.) leaving the resource in an unknown and inconsistent state.

Race conditions often result in unexpected behavior of a program and/or corrupt data.

These sensitive parts of code that can be executed by multiple threads concurrently and may result in race conditions are called critical sections. A critical section may refer to a single block of code, but it also refers to multiple accesses to the same data variable or resource from multiple functions.

Locks can protect shared variables, like data or program state.
Locks can protect shared resources, like access to a socket or file.

A mutex lock can be used to ensure that only one thread at a time executes a critical section of code at a time, while all other threads trying to execute the same code must wait until the currently executing thread is finished with the critical section and releases the lock.

Mutual exclusion: The process of allowing only one thread at a time to execute a given portion of code.
— Page 269, The Art of Concurrency, 2009.

Each thread must attempt to acquire the lock at the beginning of the critical section. If the lock has not been obtained, then a thread will acquire it and other threads must wait until the thread that acquired the lock releases it.

… mutexes (or mutual exclusion locks), which means that at most one thread may own the lock. When thread A attempts to acquire a lock held by thread B, A must wait, or block, until B releases it. If B never releases the lock, A waits forever.
— Page 25, Java Concurrency in Practice, 2006.

If the lock has not been acquired, we might refer to it as being in the “unlocked” state. Whereas if the lock has been acquired, we might refer to it as being in the “locked” state.

Unlocked: The lock has not been acquired and can be acquired by the next thread that makes an attempt.
Locked: The lock has been acquired by one thread and any thread that makes an attempt to acquire it must wait until it is released.

Locks are created in the unlocked state.

Now that we know what a mutex lock is, let’s take a look at how we can use it in Python.

Locks in Python

Python provides a mutual exclusion lock via the threading.Lock class.

An instance of the lock can be created and then acquired by threads before accessing a critical section, and released after the critical section.

For example:

...

# create a lock

lock = Lock()

# acquire the lock

lock.acquire()

# ...

# release the lock

lock.release()

Only one thread can have the lock at any time. If a thread does not release an acquired lock, it cannot be acquired again.

The thread attempting to acquire the lock will block until the lock is acquired, such as if another thread currently holds the lock then releases it.

A primitive lock is in one of two states, “locked” or “unlocked”. It is created in the unlocked state.
— LOCK OBJECTS, THREADING – THREAD-BASED PARALLELISM.

We can attempt to acquire the lock without blocking by setting the “blocking” argument to False. If the lock cannot be acquired, a value of False is returned.

...

# acquire the lock without blocking

lock.acquire(blocking=false)

We can also attempt to acquire the lock with a timeout, that will wait the set number of seconds to acquire the lock before giving up. If the lock cannot be acquired, a value of False is returned.

...

# acquire the lock with a timeout

lock.acquire(timeout=10)

We can also use the lock via the context manager protocol via the with statement, allowing the critical section to be a block within the usage of the lock and for the lock to be released automatically once the block has completed.

For example:

...

# create a lock

lock = Lock()

# acquire the lock

with lock:

# ...

This is the preferred usage as it makes it clear where the protected code begins and ends, and ensures that the lock is always released, even if there is an exception or error within the critical section.

We can also check if the lock is currently acquired by a thread via the locked() function.

...

# check if a lock is currently acquired

if lock.locked():

# ...

You can learn more about mutex locks in this tutorial:

How to Use a Mutex Lock in Python

A threading.RLock class provides a lock that can be acquired multiple times by the same thread, unlike the threading.Lock. This is called a reentrant lock.

Now that we are familiar with Lock, let’s take a look at Semaphore.

Run loops using all CPUs, download your FREE book to learn how.

What is Semaphore

A semaphore is a concurrency primitive that allows a limit on the number of threads that can acquire a lock protecting a critical section.

This is one of the oldest synchronization primitives in the history of computer science, invented by the early Dutch computer scientist Edsger W. Dijkstra
— SEMAPHORE OBJECTS

It is an extension of a mutual exclusion (mutex) lock that adds a count for the number of threads that can acquire the lock before additional threads will block. Once full, new threads can only acquire a position on the semaphore once an existing thread holding the semaphore releases a position.

Internally, the semaphore maintains a counter protected by a mutex lock that is incremented each time the semaphore is acquired and decremented each time it is released.

When a semaphore is created, the upper limit on the counter is set. If it is set to be 1, then the semaphore will operate like a mutex lock.

Semaphore: A synchronization object that has an associated nonnegative count. Two operations that are defined on a semaphore are known as wait (if the count is nonzero, decrement count; otherwise, block until count is positive) and post (increment count).
— Page 272, The Art of Concurrency, 2009.

A semaphore provides a useful concurrency tool for limiting the number of threads that can access a resource concurrently. Some examples include:

Limiting concurrent socket connections to a server.
Limiting concurrent file operations on a hard drive.
Limiting concurrent calculations.

Now that we know what a semaphore is, let’s look at how we might use it in Python.

Semaphores in Python

Python provides a semaphore via the threading.Semaphore class.

The threading.Semaphore instance must be configured when it is created to set the limit on the internal counter. This limit will match the number of concurrent threads that can hold the semaphore.

For example, we might want to set it to 100:

...

# create a semaphore with a limit of 100

semaphore = Semaphore(100)

In this implementation, each time the semaphore is acquired, the internal counter is decremented. Each time the semaphore is released, the internal counter is incremented. The semaphore cannot be acquired if the semaphore has no available positions in which case, threads attempting to acquire it must block until a position becomes available.

The semaphore can be acquired by calling the acquire() function, for example:

...

# acquire the semaphore

semaphore.acquire()

By default, it is a blocking call, which means that the calling thread will block until a position becomes available on the semaphore.

The “blocking” argument can be set to False in which case, if a position is not available on the semaphore, the thread will not block and the function will return immediately, returning a False value indicating that the semaphore was not acquired or a True value if a position could be acquired.

...

# acquire the semaphore without blocking

semaphore.acquire(blocking=False)

The “timeout” argument can be set to a number of seconds that the calling thread is willing to wait for a position on the semaphore if one is not available, before giving up. Again, the acquire() function will return a value of True if a position could be acquired or False otherwise.

...

# acquire the semaphore with a timeout

semaphore.acquire(timeout=10)

Once acquired, the semaphore can be released again by calling the release() function.

...

# release the semaphore

semaphore.release()

More than one position can be made available by calling release and setting the “n” argument to an integer number of positions to release on the semaphore.

This might be helpful if it is known a thread has died without correctly releasing the semaphore, or if one thread acquires the same semaphore more than once.

Do not use this argument unless you have a clear use case, it is likely to get you into trouble with a semaphore left in an inconsistent state.

...

# release three positions on the semaphore

semaphore.release(n=3)

Finally, the threading.Semaphore class supports usage via the context manager, which will automatically acquire and release the semaphore for you. As such it is the preferred usage, if appropriate for your program.

For example:

...

# acquire the semaphore

with semaphore:

# ...

It is possible for the release() method to be called more times than the acquire() method, and this will not cause a problem. It is a way of increasing the capacity of the semaphore.

You can learn more about semaphores in the tutorials:

A threading.BoundedSemaphore class can be used which will check to ensure that the internal counter does not go above the initial value via calls to release(). If it does, a ValueError will be raised, treating it as an error condition.

Now that we are familiar with the Lock and Semaphore, let’s compare and contrast each.

Download Now: Free Threading PDF Cheat Sheet

Comparison of Lock vs Semaphore

Now that we are familiar with the Lock and Semaphore classes, let’s review their similarities and differences.

Similarities Between Lock and Semaphore

The Lock and Semaphore classes are very similar, let’s review some of the most important similarities.

Both are concurrency primitives.
Both protect a critical section.
Both support the same interface.

Both Are Concurrency Primitives

Both locks and semaphores are concurrency primitives.

As such, they are building blocks used on concurrency programming, along with latches, barriers, events, conditions and more.

Although both primitives, a lock is perhaps simpler and therefore a lower-level.

Both Protect Critical Sections

Both locks and semaphores are used to protect a critical section from concurrent access by multiple threads.

This is the central responsibility of both mechanisms.

The critical section may be a shared variable, data, or resource.

Although they both protect critical sections, locks are more limiting than semaphores.

Both Have The Same Interface

Both locks and semaphores support the same interface.

The Python implementation offers near identical usage both classes, including:

Both support acquire() and release() methods.
The acquire() method supports blocking, non-blocking and a timeout.
Both support a context manager interface to acquire then release automatically.

Although mostly similar, there are a few minor differences.

Differences Between Lock and Semaphore

The Lock and Semaphore are also quite different, let’s review some of the most important differences.

Access: Lock is mutually exclusive access, Semaphore supports controlled access.
Complexity: Lock is simpler, Semaphore is more complex.
Interface: Locks offer no configuration, Semaphores require configuration.

Difference in Access

A mutex only allows one thread at a time to acquire and hold a lock.

All other threads must block and wait for access.

Whereas a semaphore allows a fixed number of threads to acquire and hold a position on the semaphore. It may be one, in which case the semaphore acts like a mutex, or it may be more than once.

This is the central difference between the lock and the semaphore.

Difference in Complexity

A mutex lock is simple.

It serializes access so that a protected section can only be executed by one thread at a time. The protected critical section becomes mutually exclusive. If the lock is acquired, threads block and wait for access.

A semaphore is more complex.

It serializes access to a critical section, but allows access to a specified number of threads concurrently, which may be one or more.

If the semaphore only allows a single thread to access the critical section, it is called a binary semaphore and will operate exactly like a mutex lock. This shows that the capabilities of a semaphore subsume the capabilities of a mutex.

A degenerate case of a counting semaphore is a binary semaphore, a Semaphore with an initial count of one. A binary semaphore can be used as a mutex with nonreentrant locking semantics; whoever holds the sole permit holds the mutex.
— Page 99, Java Concurrency in Practice, 2006.

If the semaphore is full, threads block and wait for access. There may be one or more positions available on the semaphore and one or more threads waiting. Therefore, internally the semaphore makes use of a threading.Condition to ensure that waiting threads are notified.

The added complexity of the semaphore allows additional use cases, such as using the semaphore as a binary latch and using the semaphore as a thread-safe counter.

Difference in Interface

A mutex lock requires no configuration.

It is created and used as-is.

A semaphore is configured.

When constructed, the user must specify the initial count for the semaphore. This indicates the number of positions that may be held on the semaphore, such as the number of concurrent threads.

Additionally, when a position on the semaphore is released, a user may specify the number of positions to release, although defaults to 1.

Summary of Differences

It may help to summarize the differences between Lock and Semaphore.

Lock

Simpler in implementation and usage.
Less access to critical sections or shared resources.
Cannot be configured.

Semaphore

More complex in implementation and usage.
May provide more access to a critical section or shared resource.
Requires configuration.

Free Python Threading Course

Download your FREE threading PDF cheat sheet and get BONUS access to my free 7-day crash course on the threading API.

Discover how to use the Python threading module including how to create and start new threads and how to use a mutex locks and semaphores

Learn more

How to Choose Lock or Semaphore

When should you use Lock and when should you use Semaphore, let’s review some useful suggestions.

When to Use Lock

Use a lock when you require mutually exclusive access to a shared block of code, variable, resource or data in your program.

That is, when the critical section must only be accessed by one thread at a time.

Don’t Use Lock for …

Do not use a lock when your code is not subject to a race condition, e.g. is immutable. In this case, you do not have a critical section and do not require a lock or a semaphore.

Do not use a lock when you require access to a critical section by more than one thread at a time, e.g. not mutually exclusive access.

When to Use the Semaphore

Use a semaphore when you require access to a critical section or resource by more than one thread.

Don’t Use Semaphore for …

Do not use a semaphore when your code is not subject to a race condition, e.g. is immutable. In this case, you do not have a critical section and do not require a lock or a semaphore.

Do not use a semaphore when your critical section must be accessed by one thread at a time, e.g. mutually exclusive. In this case use a lock. Do not use a binary semaphore as it is too complex, e.g. overkill.

Overwhelmed by the python concurrency APIs?
Find relief, download my FREE Python Concurrency Mind Maps

Takeaways

You now know the difference between Lock and Semaphore and when to use each.

Do you have any questions about the difference between Lock and Semaphore in Python?
Ask your questions in the comments below and I will do my best to answer.

Photo by Bradley Dunn on Unsplash