Skip to content

gh-97847: Fix misplaced or duplicated function descriptions in the multiprocessing documentation #31482

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
geryogam wants to merge 10 commits into from
Closed

gh-97847: Fix misplaced or duplicated function descriptions in the multiprocessing documentation #31482

Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
fa51ede
Update multiprocessing.rst
geryogam Feb 21, 2022
465e589
Update multiprocessing.rst
geryogam Feb 23, 2022
51132fa
Update multiprocessing.rst
geryogam Feb 24, 2022
7993362
Update multiprocessing.rst
geryogam Feb 25, 2022
98d08f0
Update multiprocessing.rst
geryogam Feb 25, 2022
1054ff1
Update multiprocessing.rst
geryogam Feb 25, 2022
cc1a30b
Update multiprocessing.rst
geryogam Feb 25, 2022
1012e2c
Update multiprocessing.rst
geryogam Feb 25, 2022
97a716d
Update Doc/library/multiprocessing.rst
geryogam Oct 4, 2022
109bb13
Merge branch 'main' into patch-6
arhadthedev Mar 4, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
126 changes: 32 additions & 94 deletions Doc/library/multiprocessing.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1470,75 +1470,9 @@ object -- see :ref:`multiprocessing-managers`.
Shared :mod:`ctypes` Objects
~~~~~~~~~~~~~~~~~~~~~~~~~~~~

It is possible to create shared objects using shared memory which can be
It is possible to create shared :mod:`ctypes` objects from shared memory which can be
inherited by child processes.

.. function:: Value(typecode_or_type, *args, lock=True)

Return a :mod:`ctypes` object allocated from shared memory. By default the
return value is actually a synchronized wrapper for the object. The object
itself can be accessed via the *value* attribute of a :class:`Value`.

*typecode_or_type* determines the type of the returned object: it is either a
ctypes type or a one character typecode of the kind used by the :mod:`array`
module. *\*args* is passed on to the constructor for the type.

If *lock* is ``True`` (the default) then a new recursive lock
object is created to synchronize access to the value. If *lock* is
a :class:`Lock` or :class:`RLock` object then that will be used to
synchronize access to the value. If *lock* is ``False`` then
access to the returned object will not be automatically protected
by a lock, so it will not necessarily be "process-safe".

Operations like ``+=`` which involve a read and write are not
atomic. So if, for instance, you want to atomically increment a
shared value it is insufficient to just do ::

counter.value += 1

Assuming the associated lock is recursive (which it is by default)
you can instead do ::

with counter.get_lock():
counter.value += 1

Note that *lock* is a keyword-only argument.

.. function:: Array(typecode_or_type, size_or_initializer, *, lock=True)

Return a ctypes array allocated from shared memory. By default the return
value is actually a synchronized wrapper for the array.

*typecode_or_type* determines the type of the elements of the returned array:
it is either a ctypes type or a one character typecode of the kind used by
the :mod:`array` module. If *size_or_initializer* is an integer, then it
determines the length of the array, and the array will be initially zeroed.
Otherwise, *size_or_initializer* is a sequence which is used to initialize
the array and whose length determines the length of the array.

If *lock* is ``True`` (the default) then a new lock object is created to
synchronize access to the value. If *lock* is a :class:`Lock` or
:class:`RLock` object then that will be used to synchronize access to the
value. If *lock* is ``False`` then access to the returned object will not be
automatically protected by a lock, so it will not necessarily be
"process-safe".

Note that *lock* is a keyword only argument.

Note that an array of :data:`ctypes.c_char` has *value* and *raw*
attributes which allow one to use it to store and retrieve strings.


The :mod:`multiprocessing.sharedctypes` module
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

.. module:: multiprocessing.sharedctypes
:synopsis: Allocate ctypes objects from shared memory.

The :mod:`multiprocessing.sharedctypes` module provides functions for allocating
:mod:`ctypes` objects from shared memory which can be inherited by child
processes.

.. note::

Although it is possible to store a pointer in shared memory remember that
Expand Down Expand Up @@ -1584,7 +1518,7 @@ processes.
process-safe synchronization wrapper may be returned instead of a raw ctypes
array.

If *lock* is ``True`` (the default) then a new lock object is created to
If *lock* is ``True`` (the default) then a new recursive lock object is created to
synchronize access to the value. If *lock* is a
:class:`~multiprocessing.Lock` or :class:`~multiprocessing.RLock` object
then that will be used to synchronize access to the
Expand All @@ -1600,7 +1534,7 @@ processes.
process-safe synchronization wrapper may be returned instead of a raw ctypes
object.

If *lock* is ``True`` (the default) then a new lock object is created to
If *lock* is ``True`` (the default) then a new recursive lock object is created to
synchronize access to the value. If *lock* is a :class:`~multiprocessing.Lock` or
:class:`~multiprocessing.RLock` object then that will be used to synchronize access to the
value. If *lock* is ``False`` then access to the returned object will not be
Expand All @@ -1609,6 +1543,21 @@ processes.

Note that *lock* is a keyword-only argument.

Operations like ``+=`` which involve a read and write are not
atomic. So if, for instance, you want to atomically increment a
shared value it is insufficient to just do ::

counter.value += 1

Assuming the associated lock is recursive (which it is by default)
you can instead do ::

with counter.get_lock():
counter.value += 1

.. module:: multiprocessing.sharedctypes
:synopsis: Allocate ctypes objects from shared memory.

.. function:: copy(obj)

Return a ctypes object allocated from shared memory which is a copy of the
Expand Down Expand Up @@ -1648,8 +1597,7 @@ MyStruct(4, 6) RawValue(MyStruct, 4, 6)
Below is an example where a number of ctypes objects are modified by a child
process::

from multiprocessing import Process, Lock
from multiprocessing.sharedctypes import Value, Array
from multiprocessing import Process, Lock, Value, Array
from ctypes import Structure, c_double

class Point(Structure):
Expand Down Expand Up @@ -2145,9 +2093,9 @@ demonstrates a level of control over the synchronization.
a new shared object -- see documentation for the *method_to_typeid*
argument of :meth:`BaseManager.register`.

If an exception is raised by the call, then is re-raised by
If an exception is raised by the call then it is re-raised by
:meth:`_callmethod`. If some other exception is raised in the manager's
process then this is converted into a :exc:`RemoteError` exception and is
process then it is converted into a :exc:`RemoteError` exception and is
raised by :meth:`_callmethod`.

Note in particular that an exception will be raised if *methodname* has
Expand Down Expand Up @@ -2195,13 +2143,10 @@ any proxies referring to it.
Process Pools
~~~~~~~~~~~~~

.. module:: multiprocessing.pool
:synopsis: Create pools of processes.

One can create a pool of processes which will carry out tasks submitted to it
with the :class:`Pool` class.

.. class:: Pool([processes[, initializer[, initargs[, maxtasksperchild [, context]]]]])
.. class:: Pool([processes[, initializer[, initargs[, maxtasksperchild]]]])

A process pool object which controls a pool of worker processes to which jobs
can be submitted. It supports asynchronous results with timeouts and
Expand All @@ -2218,12 +2163,6 @@ with the :class:`Pool` class.
unused resources to be freed. The default *maxtasksperchild* is ``None``, which
means worker processes will live as long as the pool.

*context* can be used to specify the context used for starting
the worker processes. Usually a pool is created using the
function :func:`multiprocessing.Pool` or the :meth:`Pool` method
of a context object. In both cases *context* is set
appropriately.

Note that the methods of the pool object should only be called by
the process which created the pool.

Expand All @@ -2240,17 +2179,14 @@ with the :class:`Pool` class.
.. versionadded:: 3.2
*maxtasksperchild*

.. versionadded:: 3.4
*context*

.. note::

Worker processes within a :class:`Pool` typically live for the complete
duration of the Pool's work queue. A frequent pattern found in other
systems (such as Apache, mod_wsgi, etc) to free resources held by
systems (such as Apache, mod_wsgi, etc.) to free resources held by
workers is to allow a worker within a pool to complete only a set
amount of work before being exiting, being cleaned up and a new
process spawned to replace the old one. The *maxtasksperchild*
amount of work before being exited, cleaned up, and replaced by a new
spawned process. The *maxtasksperchild*
argument to the :class:`Pool` exposes this ability to the end user.

.. method:: apply(func[, args[, kwds]])
Expand Down Expand Up @@ -2368,11 +2304,13 @@ with the :class:`Pool` class.
:ref:`typecontextmanager`. :meth:`~contextmanager.__enter__` returns the
pool object, and :meth:`~contextmanager.__exit__` calls :meth:`terminate`.

.. module:: multiprocessing.pool
:synopsis: Create pools of processes.

.. class:: AsyncResult

The class of the result returned by :meth:`Pool.apply_async` and
:meth:`Pool.map_async`.
The class of the result returned by :meth:`Pool.apply_async`,
:meth:`Pool.map_async`, and :meth:`Pool.starmap_async`.

.. method:: get([timeout])

Expand Down Expand Up @@ -2430,7 +2368,7 @@ Listeners and Clients
.. module:: multiprocessing.connection
:synopsis: API for dealing with sockets.

Usually message passing between processes is done using queues or by using
Usually message passing between processes is done using queues or
:class:`~Connection` objects returned by
:func:`~multiprocessing.Pipe`.

Expand Down Expand Up @@ -2771,10 +2709,10 @@ worker threads rather than worker processes.
*processes* is the number of worker threads to use. If *processes* is
``None`` then the number returned by :func:`os.cpu_count` is used.

If *initializer* is not ``None`` then each worker process will call
If *initializer* is not ``None`` then each worker thread will call
``initializer(*initargs)`` when it starts.

Unlike :class:`Pool`, *maxtasksperchild* and *context* cannot be provided.
Unlike :class:`Pool`, *maxtasksperchild* cannot be provided.

.. note::

Expand Down