Multiprocessing Semaphore in Python

Last Updated on March 17, 2023

You can use a semaphore for processes via multiprocessing.Semaphore class.

In this tutorial you will discover how to use a semaphore for processes in Python.

Let’s get started.

Table of Contents

Need for a Semaphore

A process is a running instance of a computer program.

Every Python program is executed in a Process, which is a new instance of the Python interpreter. This process has the name MainProcess and has one thread used to execute the program instructions called the MainThread. Both processes and threads are created and managed by the underlying operating system.

Sometimes we may need to create new child processes in our program in order to execute code concurrently.

Python provides the ability to create and manage new processes via the multiprocessing.Process class.

You can learn more about multiprocessing in the tutorial:

Multiprocessing in Python: The Complete Guide

In concurrent programming, we may need to limit the number of concurrent processes that can perform an action simultaneously, such as execute a critical section, perform a calculation, or operate an external resource.

This could be achieved with a mutual exclusion lock and a counter that would have to be manually maintained.

An alternative is to use a semaphore.

What is a semaphore and how can we use it with processes in Python?

Run loops using all CPUs, download your FREE book to learn how.

What is a Semaphore

A semaphore is a concurrency primitive that allows a limit on the number of processes (or threads) that can acquire a lock protecting a critical section.

It is an extension of a mutual exclusion (mutex) lock that adds a count for the number of processes that can acquire the lock before additional processes will block. Once full, new processes can only acquire access on the semaphore once an existing process holding the semaphore releases access.

Internally, the semaphore maintains a counter protected by a mutex lock that is incremented each time the semaphore is acquired and decremented each time it is released.

When a semaphore is created, the upper limit on the counter is set. If it is set to be 1, then the semaphore will operate like a mutex lock.

A semaphore provides a useful concurrency tool for limiting the number of processes that can access a resource concurrently. Some examples include:

Limiting concurrent socket connections to a server.
Limiting concurrent file operations on a hard drive.
Limiting concurrent calculations.

Semaphores are also used with threads, you can learn more about them in this tutorial:

How to Use a Semaphore in Python

Now that we know what a semaphore is, let’s look at how we might use it in Python.

Download Now: Free Multiprocessing PDF Cheat Sheet

How to Use a Semaphore for Processes

Python provides a semaphore for processes via the multiprocessing.Semaphore class.

The multiprocessing.Semaphore instance must be configured when it is created to set the limit on the internal counter. This limit will match the number of concurrent processes that can hold the semaphore.

For example, we might want to set it to 100:

...

# create a semaphore with a limit of 100

semaphore = multiprocessing.Semaphore(100)

In this implementation, each time the semaphore is acquired, the internal counter is decremented. Each time the semaphore is released, the internal counter is incremented. The semaphore cannot be acquired if the semaphore has no available access in which case, processes attempting to acquire it must block until access becomes available.

The semaphore can be acquired by calling the acquire() function, for example:

...

# acquire the semaphore

semaphore.acquire()

By default, it is a blocking call, which means that the calling process will block until access becomes available on the semaphore.

The “block” argument can be set to False in which case, if access is not available on the semaphore, the process will not block and the function will return immediately, returning a False value indicating that the semaphore was not acquired or a True value if access could be acquired.

...

# acquire the semaphore without blocking

semaphore.acquire(block=False)

The “timeout” argument can be set to a number of seconds that the calling process is willing to wait for access to the semaphore if one is not available, before giving up. Again, the acquire() function will return a value of True if access could be acquired or False otherwise.

...

# acquire the semaphore with a timeout

semaphore.acquire(timeout=10)

Once acquired, the semaphore can be released again by calling the release() function.

...

# release the semaphore

semaphore.release()

More than one position can be made available by calling release and setting the “n” argument to an integer number of positions to release on the semaphore.

This might be helpful if it is known a process has died without correctly releasing the semaphore, or if one process acquires the same semaphore more than once.

Do not use this argument unless you have a clear use case, it is likely to get you into trouble with a semaphore left in an inconsistent state.

...

# release three positions on the semaphore

semaphore.release(n=3)

Finally, the multiprocessing.Semaphore class supports usage via the context manager, which will automatically acquire and release the semaphore for you. As such it is the preferred usage, if appropriate for your program.

For example:

...

# acquire the semaphore

with semaphore:

# ...

Now that we know how to use the multiprocessing.Semaphore in Python, let’s look at a worked example.

Free Python Multiprocessing Course

Download your FREE multiprocessing PDF cheat sheet and get BONUS access to my free 7-day crash course on the multiprocessing API.

Discover how to use the Python multiprocessing module including how to create and start child processes and how to use a mutex locks and semaphores.

Learn more

Example of Using a Semaphore

We can explore how to use a multiprocessing.Semaphore with a worked example.

We will develop an example with a suite of processes but a limit on the number of processes that can perform an action simultaneously. A semaphore will be used to limit the number of concurrent processes which will be less than the total number of processes, allowing some processes to block, wait for access, then be notified and acquire access.

First, we can define a target task function that takes the shared semaphore and a unique integer as arguments. The function will attempt to acquire the semaphore, and once access is acquired it will simulate some processing by generating a random number and blocking for a moment, then report its data and release the semaphore.

The complete task() function is listed below.

# target function

def task(semaphore, number):

# attempt to acquire the semaphore

with semaphore:

# simulate computational effort

value = random()

sleep(value)

# report result

print(f'Process {number} got {value}')

The main process will first create the multiprocessing.Semaphore instance and limit the number of concurrent processes to 2.

...

# create the shared semaphore

semaphore = Semaphore(2)

Next, we will create and start 10 processes and pass each the shared semaphore instance and a unique integer to identify the process.

We can implement this via a list comprehension, creating a list of ten configured multiprocessing.Process instances.

...

# create processes

processes = [Process(target=task, args=(semaphore, i)) for i in range(10)]

Next, we can start all of the processes.

...

# start child processes

for process in processes:

process.start()

Finally, we can wait for all of the new child processes to terminate.

...

# wait for child processes to finish

for process in processes:

process.join()

Tying this together, the complete example is listed below.

# SuperFastPython.com

# example of using a semaphore

from time import sleep

from random import random

from multiprocessing import Process

from multiprocessing import Semaphore

# target function

def task(semaphore, number):

# attempt to acquire the semaphore

with semaphore:

# simulate computational effort

value = random()

sleep(value)

# report result

print(f'Process {number} got {value}')

# entry point

if __name__ == '__main__':

# create the shared semaphore

semaphore = Semaphore(2)

# create processes

processes = [Process(target=task, args=(semaphore, i)) for i in range(10)]

# start child processes

for process in processes:

process.start()

# wait for child processes to finish

for process in processes:

process.join()

Running the example first creates the shared semaphore instance then starts ten child processes.

All ten processes attempt to acquire the semaphore, but only two processes are granted access at a time. The processes on the semaphore do their work and release the semaphore when they are done, at random intervals.

Each release of the semaphore (via the context manager) allows another process to acquire access and perform its simulated calculation, all the while allowing only two of the processes to be running within the critical section at any one time, even though all ten processes are executing their run methods.

Note, your specific values will differ given the use of random numbers.