mpi - mpi4py Split_type using openmpi's OMPI_COMM_TYPE_SOCKET

Question

Is it possible to split a communicator using openmpi's OMPI_COMM_TYPE_SOCKET in mpi4py?

I've verified this works:

from mpi4py import MPI

comm = MPI.COMM_WORLD

sharedcomm = comm.Split_type(MPI.COMM_TYPE_SHARED)

but this does not:

socketcomm = comm.Split_type(MPI.OMPI_COMM_TYPE_SOCKET)

nor does this:

socketcomm = comm.Split_type(MPI.COMM_TYPE_SOCKET)

I've looked at the docs but I can't find anything about this.

question from:https://stackoverflow.com/questions/65888672/mpi4py-split-type-using-openmpis-ompi-comm-type-socket

Answer 1

mpi4py only provides a wrapper around standard MPI features. OMPI_COMM_TYPE_SOCKET is an Open MPI specific split type. You may still use it in mpi4py if you know its numeric value as it is just a member of a C enum:

/*
 * Communicator split type constants.
 * Do not change the order of these without also modifying mpif.h.in
 * (see also mpif-common.h.fin).
 */
enum {
  MPI_COMM_TYPE_SHARED,
  OMPI_COMM_TYPE_HWTHREAD,
  OMPI_COMM_TYPE_CORE,
  OMPI_COMM_TYPE_L1CACHE,
  OMPI_COMM_TYPE_L2CACHE,
  OMPI_COMM_TYPE_L3CACHE,
  OMPI_COMM_TYPE_SOCKET,  // here
  OMPI_COMM_TYPE_NUMA,
  OMPI_COMM_TYPE_BOARD,
  OMPI_COMM_TYPE_HOST,
  OMPI_COMM_TYPE_CU,
  OMPI_COMM_TYPE_CLUSTER
};
#define OMPI_COMM_TYPE_NODE MPI_COMM_TYPE_SHARED

Being a member of an enum means that the actual numerical value of OMPI_COMM_TYPE_SOCKET depends on its position in the enum and hence may differ from one release of Open MPI to another. You have several options here.

Hardcode the value

This is the simplest option. Open mpi.h (ompi_info --path incdir gives you its location), count the position of OMPI_COMM_TYPE_SOCKET in the enclosing enum starting with 0 for MPI_COMM_TYPE_SHARED and hardcode the value. The code may break with releases of Open MPI different from yours.

Parse mpi.h

Read mpi.h, search for enum definitions and find the one containing OMPI_COMM_TYPE_SOCKET. Provided that MPI_COMM_TYPE_SHARED is 0, the value of OMPI_COMM_TYPE_SOCKET is its 0-based index in the sequence of enum values. This depends a lot on the code in mpi.h having a specific format and can easily break if that changes.

Parse mpif.h

The Fortran interface is easier to parse as there the value is defined as:

parameter (OMPI_COMM_TYPE_SOCKET=6)

This is easily parsable with a simple regular expression. The problem is that recent versions of Open MPI split mpif.h over a couple of files that are then included from mpif.h and currently the value is in mpif-constants.h. So you may need to parse the include statements and recurse into the files they reference. Note that those are Fortran include statements and not preprocessor #include directives.

Code generation

Write a small C program that outputs the value of OMPI_COMM_TYPE_SOCKET to a Python file and run it as part of your program's setup procedure. Something like:

#include <stdio.h>
#include <mpi.h>

int main (int argc, char **argv)
{
    if (argc != 2)
    {
        printf("Usage: mkompimod /path/to/module.py
");
        return 1;
    }
    FILE *fh = fopen(argv[1], "w");
    if (fh != NULL) {
        fprintf(fh, "COMM_TYPE_SOCKET = %d
", OMPI_COMM_TYPE_SOCKET);
        fclose(fh);
    }
    return 0;
}

Put that in a file named mkompimod.c. Compile with mpicc -o mkompimod mkompimod.c and run with mkompimod /path/to/ompi.py to create a Python file ompi.py with the value of OMPI_COMM_TYPE_SOCKET. Import it and use it in the call to comm.Split_type():

import ompi

socketcomm = comm.Split_type(ompi.COMM_TYPE_SOCKET)

Write a Python module in C

That's a bit involved, but you can write a C module that includes mpi.h and exports the value of OMPI_COMM_TYPE_SOCKET as a Python constant. Consult the Python documentation on how to write extensions in C.

Use the CFFI module

CFFI lets you build Python modules that wrap C libraries and writes all the glue code for you. Put the following in a file named ompi_build.py:

from cffi import FFI

ffi = FFI()
ffi.set_source("ompi", r"#include <mpi.h>")
ffi.cdef(
   r"""
   const int OMPI_COMM_TYPE_HWTHREAD;
   ... more constants here ...
   const int OMPI_COMM_TYPE_SOCKET;
   ... even more constants here ...
   """
)

if __name__ == "__main__":
   ffi.compile(verbose=True)

Run like this:

$ CC=mpicc python ompi_build.py

This will create the C module ompi.c and compile it into a loadable DSO. You may then import it and access the constant like this:

from ompi.lib import OMPI_COMM_TYPE_SOCKET

socketcomm = comm.Split_type(OMPI_COMM_TYPE_SOCKET)

CFFI provides integration with Python's distutils and you can have it automatically build the C module as part of the setup process.

Use Cython

That's what mpi4py itself is written in. It blends C and Python into a single cryptic syntax. Read the source code. Try to figure out what's going on and how to write something yourself. Can't help you there.

---

Whichever path you choose, keep in mind that all this pertains to the system on which the program will be running, not only the system on which the program will be developed.