pool.py - Multiprocessing

An implementation of three different types of pools:

  • An MPI pool borrowed from emcee. This pool passes Python objects back and forth to the workers and communicates once per task.
  • A multiprocessing for local parallelization, borrowed from emcee
  • A serial pool, which uses the built-in map function
class everest.pool.MPIPool(comm=None, loadbalance=False, debug=False, wait_on_start=True, exit_on_end=True, cores_per_task=1, **kwargs)

A pool that distributes tasks over a set of MPI processes. MPI is an API for distributed memory parallelism. This pool will let you run emcee without shared memory, letting you use much larger machines with emcee.

The pool only support the map() method at the moment because this is the only functionality that emcee needs. That being said, this pool is fairly general and it could be used for other purposes.

Contributed by Joe Zuntz.

Parameters:
  • comm – (optional) The mpi4py communicator.
  • loadbalance – (optional) if True and ntask > Ncpus, tries to loadbalance by sending out one task to each cpu first and then sending out the rest as the cpus get done.
bcast(*args, **kwargs)

Equivalent to mpi4py bcast() collective operation.

close()

Just send a message off to all the pool members which contains the special _close_pool_message sentinel.

static enabled()
map(function, tasks)

Like the built-in map() function, apply a function to all of the values in a list and return the list of results.

Parameters:
  • function – The function to apply to the list.
  • tasks – The list of elements.
wait()

If this isn’t the master process, wait for instructions.

class everest.pool.MultiPool(processes=None, initializer=None, initargs=(), **kwargs)

This is simply emcee’s InterruptiblePool.

A modified version of multiprocessing.pool.Pool that has better behavior with regard to KeyboardInterrupts in the map() method.

Contributed by Peter K. G. Williams <peter@newton.cx>.

Parameters:
  • processes – (optional) The number of worker processes to use; defaults to the number of CPUs.
  • initializer – (optional) Either None, or a callable that will be invoked by each worker process when it starts.
  • initargs – (optional) Arguments for initializer; it will be called as initializer(*initargs).
  • kwargs – (optional) Extra arguments. Python 2.7 supports a maxtasksperchild parameter.
static enabled()
map(func, iterable, chunksize=None)

Equivalent of map() built-in, without swallowing KeyboardInterrupt.

Parameters:
  • func – The function to apply to the items.
  • iterable – An iterable of items that will have func applied to them.
class everest.pool.SerialPool(**kwargs)
static enabled()
map(function, iterable)
wait()
everest.pool.Pool(pool='AnyPool', **kwargs)

Chooses between the different pools. If pool == 'AnyPool', chooses based on availability.