Last Updated on July 29, 2023

You can use a semaphore in Python by threading.Semaphore class.

In this tutorial you will discover how to use a semaphore for concurrent programming.

Let’s get started.

Need for a Semaphore

A thread is a thread of execution in a computer program.

Every Python program has at least one thread of execution called the main thread. Both processes and threads are created and managed by the underlying operating system.

Sometimes we may need to create additional threads in our program in order to execute code concurrently.

Python provides the ability to create and manage new threads via the threading module and the threading.Thread class.

You can learn more about Python threads in the guude:

• Threading in Python: The Complete Guide

In concurrent programming, we may need to limit the number of concurrent threads that can perform an action simultaneously, such as execute a critical section, perform a calculation, or operate an external resource.

This could be achieved with a mutual exclusion lock and a counter that would have to be manually maintained.

An alternative is to use a semaphore.

What is a semaphore and how can we use it in Python?

What is a Semaphore

A semaphore is a concurrency primitive that allows a limit on the number of threads that can acquire a lock protecting a critical section.

It is an extension of a mutual exclusion (mutex) lock that adds a count for the number of threads that can acquire the lock before additional threads will block. Once full, new threads can only acquire a position on the semaphore once an existing thread holding the semaphore releases a position.

Internally, the semaphore maintains a counter initialized to a provided value, protected by a mutex lock that is decremented each time the semaphore is acquired and incremented each time it is released.

A semaphore manages an internal counter which is decremented by each acquire() call and incremented by each release() call. The counter can never go below zero; when acquire() finds that it is zero, it blocks, waiting until some other thread calls release().

— threading — Thread-based parallelism

When a semaphore is created, the upper limit on the counter is set. If it is set to 1, then the semaphore will operate like a mutex lock.

A semaphore provides a useful concurrency tool for limiting the number of threads that can access a resource concurrently. Some examples include:

• Limiting concurrent socket connections to a server.
• Limiting concurrent file operations on a hard drive.
• Limiting concurrent calculations.

Now that we know what a semaphore is, let’s look at how we might use it in Python.

How to Use a Semaphore

Python provides a semaphore via the threading.Semaphore class.

The threading.Semaphore instance must be configured when it is created to set the limit on the internal counter. This limit will match the number of concurrent threads that can hold the semaphore.

For example, we might want to set it to 100:

In this implementation, each time the semaphore is acquired, the internal counter is decremented. Each time the semaphore is released, the internal counter is incremented. The semaphore cannot be acquired if the semaphore has no available positions in which case, threads attempting to acquire it must block until a position becomes available.

The semaphore can be acquired by calling the acquire() function, for example:

By default, it is a blocking call, which means that the calling thread will block until a position becomes available on the semaphore.

The “blocking” argument can be set to False in which case, if a position is not available on the semaphore, the thread will not block and the function will return immediately, returning a False value indicating that the semaphore was not acquired or a True value if a position could be acquired.

The “timeout” argument can be set to a number of seconds that the calling thread is willing to wait for a position on the semaphore if one is not available, before giving up. Again, the acquire() function will return a value of True if a position could be acquired or False otherwise.

Once acquired, the semaphore can be released again by calling the release() function.

More than one position can be made available by calling release and setting the “n” argument to an integer number of positions to release on the semaphore.

This might be helpful if it is known a thread has died without correctly releasing the semaphore, or if one thread acquires the same semaphore more than once.

Do not use this argument unless you have a clear use case, it is likely to get you into trouble with a semaphore left in an inconsistent state.

Finally, the threading.Semaphore class supports usage via the context manager, which will automatically acquire and release the semaphore for you. As such it is the preferred usage, if appropriate for your program.

For example:

It is possible for the release() method to be called more times than the acquire() method, and this will not cause a problem. It is a way of increasing the capacity of the semaphore.

A threading.BoundedSemaphore class can be used which will check to ensure that the internal counter does not go above the initial value via calls to release(). If it does, a ValueError will be raised, treating it as an error condition.

Now that we know how to use the threading.Semaphore in Python, let’s look at a worked example.

Example of Using a Semaphore

We can explore how to use a threading.Semaphore with a worked example.

We will develop an example with a suite of threads but a limit on the number of threads that can perform an action simultaneously. A semaphore will be used to limit the number of concurrent threads which will be less than the total number of threads, allowing some threads to block, wait for a position, then be notified and acquire a position.

First, we can define a target task function that takes the shared semaphore and a unique integer as arguments. The function will attempt to acquire the semaphore, and once a position is acquired it will simulate some processing by generating a random number and blocking for a moment, then report its data and release the semaphore.

The complete task() function is listed below.

The main thread will first create the threading.Semaphore instance and limit the number of concurrent processing threads to 2.

Next, we will create and start 10 threads and pass each the shared semaphore instance and a unique integer to identify the thread.

The main thread will then wait for all worker threads to complete before exiting.

Tying this together, the complete example of using a semaphore is listed below.

Running the example first creates the semaphore instance then starts ten worker threads.

All ten threads attempt to acquire the semaphore, but only two threads are granted positions at a time. The threads on the semaphore do their work and release the semaphore when they are done, at random intervals.

Each release of the semaphore (via the context manager) allows another thread to acquire a position and perform its calculation, all the while allowing only two of the threads to be processed at any one time, even though all ten threads are executing their run methods.

Note, your specific values will differ given the use of random numbers.

More Help With Semaphores

If you need more help with semaphores, some of these tutorials may be interesting:

Semaphores with threads:

• Advanced Semaphore Examples
• Lock vs Semaphore in Python

Semaphores with processes:

• Multiprocessing Semaphore in Python
• Use a Semaphore in the Multiprocessing Pool

Semaphores with asyncio:

• Asyncio Semaphore in Python
• How to use an Asyncio BoundedSemaphore

Further Reading

This section provides additional resources that you may find helpful.

Python Threading Books

• Python Threading Jump-Start, Jason Brownlee (my book!)
• Threading API Interview Questions
• Threading Module API Cheat Sheet

I also recommend specific chapters in the following books:

• Python Cookbook, David Beazley and Brian Jones, 2013.
  • See: Chapter 12: Concurrency
• Effective Python, Brett Slatkin, 2019.
  • See: Chapter 7: Concurrency and Parallelism
• Python in a Nutshell, Alex Martelli, et al., 2017.
  • See: Chapter: 14: Threads and Processes

Guides

• Python Threading: The Complete Guide
• Python ThreadPoolExecutor: The Complete Guide
• Python ThreadPool: The Complete Guide

APIs

• threading - Thread-based parallelism
• queue — A synchronized queue class

References

• Thread (computing), Wikipedia.
• Process (computing), Wikipedia.

Takeaways

You now know how to use a semaphore in Python.

Do you have any questions?
Ask your questions in the comments below and I will do my best to answer.

Photo by NoName on Unsplash