%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%                                  %
% Title   : m_d.tex                %
% Subject : Maintenance manual of  %
%           Scotch                 %
%           Code explanations      %
% Author  : Francois Pellegrini    %
%                                  %
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\section{Data structure explanations}
\label{sec-data}

This section explains some of the most complex data structures
implemented in \scotch\ and \ptscotch.

\subsection{\texttt{Kgraph}}
\label{sec-data-kgraph}

The \texttt{Kgraph} structure holds all the information necessary to
compute a k-way (re)mapping of some graph onto a target architecture.
Consequently, it contains a \texttt{Graph}, defined as field
\texttt{s}, and a reference to an \texttt{Arch}, through the field
\texttt{m.archptr}, as well as two \texttt{Mapping} structures: one
for the current mapping to compute, and one to store the old mapping
from which to remap. Additional information comprise data to model the
cost of remapping, and data associated with the state and cost of the
current mapping: list of frontier vertices, load of each partition
domain, plus the execution context for multi-threading execution.

The \texttt{Graph} structure is internal to the \texttt{Kgraph}
because every new \texttt{Kgraph} contains a different graph topology
(\eg, a band graph or a coarsened graph). The \texttt{Arch} is
accessed by reference because it is constant data which can be shared
by many \texttt{Kgraph}s. For the sake of consistency, the
\texttt{grafptr} fields of each mapping \texttt{m} and \texttt{r.m}
must point to \texttt{\&s}, while their two \texttt{archptr} fields
must point to the same target architecture. This redundency is the
price to pay for lighter memory management.

\subsubsection{Mappings}

The \texttt{domnorg} field, which must contain a valid domain in the
architecture \texttt{m.archptr}, is the starting point for the k-way
mapping. This domain may be smaller than the full architecture when
parallel partitioning is performed: in this case, each process may
receive a separate subgraph and sub-architecture to work on.

Each of the two mappings has its own specificities. The current
mapping, defined as field \texttt{m}, is never incomplete: all the
cells of its \texttt{m.parttax} array are non-negative values that index
a valid domain in the domain array \texttt{m.domntab}. These domains
are all subdomains of the architecture referenced through field
\texttt{m.archptr}. More restrictively, the domains attached to
non-fixed vertices must be included in \texttt{domnorg}, which may be
smaller.

The current mapping evolves with time, according to the various
algorithms that the user can activate in the strategy string. These
algorithms will create derived \texttt{Kgraph}s (\eg, band graphs or a
coarsened graphs), to which mapping methods will be applied, before
the result is ported back to their parent \texttt{Kgraph}. Depending
on the kind of the derived graph, the \texttt{m.parttax} array may be
specific, but the \texttt{m.domntab} array will always be ported back
as is. Consequently, in order to save memory copying, the policy which
is implemented is that the derived \texttt{Kgraph} gets the pointer to
the \texttt{m.domntab} of its parent, while the latter is set to
\texttt{NULL}. The derived graph can therefore reallocate the array
whenever needed, without the risk of an old, invalid, pointer being
kept elsewhere. Then, when the processing of the derived
\texttt{Kgraph} ends, the most recent pointer is copied back to the
\texttt{m.domntab} field of the parent graph, and the
\texttt{m.parttax} array is updated accordingly, after which the
derived \texttt{Kgraph} can be destroyed without freeing the
pointer.

The old mapping, defined as field \texttt{r.m},
may contain incomplete mapping information: some of the cells of its
\texttt{r.m.parttax} array may be equal to \texttt{-1}, to indicate
that no prior mapping information is available (\eg, when the vertex
did not exist in the previous mapping). Since old mappings do not
change, the \texttt{r.m.domntab} field can be shared among all derived
\texttt{Kgraph}s. It is protected from double memory freeing by not
setting the \texttt{MAPPING\lbt FREE\lbt DOMN} flag in field
\texttt{r.m.flagval}.

\subsection{\texttt{Mapping}}
\label{sec-data-mapping}

The \texttt{Mapping} structure defines how individual vertices of a
\texttt{Graph} are mapped individually onto (parts of) an
\texttt{Arch}. A mapping is said \textit{complete} if all source graph
vertices are assigned to terminal target domains, \ie, individual
vertices of the target architecture, or \textit{partial} if at least
one of the source graph vertices is assigned to a target domain that
comprises more than one vertex. In the course of the graph mapping
process, the destination of source vertices are progressively refined,
from an initial target domain that usually describes the whole of the
target architecture, to terminal domains.

Since \texttt{ArchDom}, the data structure that describes target
architecture domains, is big and costly to handle (\eg, to compare if
two \texttt{ArchDom}s are identical), the handling of domains in
mapping is indirect: in the part array \texttt{parttax}, each vertex
is assigned an integer domain index that refers to a domain located in
the domain array \texttt{domntab}. Hence, when two graph vertices have
the same index in \texttt{parttax}, they belong to the same domain and
induce no communication cost. However, the opposite is false: two
vertices may have a different index in \texttt{parttax} and yet belong
to the same target domain. This is for instance the case when one of
the vertices is a fixed vertex that has been set to a specific
terminal domain at initialization time, and one of its neighbors is
successively mapped to smaller and smaller subdomains that eventually
amount to the same terminal domain.

In the case of a remapping, the mapping information regarding the
former placement of the vertices may be incomplete, \eg, because the
vertex did not exist before. Such a mapping is said to be
\textit{incomplete}. It is characterized by the fact that some cells
of the \texttt{parttax} array are equal to \texttt{-1}, to indicate an
unknown terminal domain number. To allow for this, the mapping must
have the \texttt{MAPPING\lbt INCOMPLETE} flag set. Incomplete mappings
are only valid when holding remapping information; new mappings being
computed must have all their \texttt{parttax} cells set with
non-negative values that point to valid domains in the
\texttt{domntab} array. New mappings can therefore only be partial or
complete.

When a mapping is initialized, all \texttt{parttax} values for
non-fixed vertices are set to index~$0$, and \texttt{domntab[0]} is
set to the root domain for the mapping. In the general case for
centralized mapping, the initial domain is equal to
\texttt{archDomFrst(archptr)}. However, when a centralized mapping
process is launched as a part of a distributed mapping process, the
initial domain may be a subset of the whole target architecture.

There is no obligation for the \texttt{domntab} array to contain only
one instance of some target domain. On the contrary, as described
above, the same domain may appear at least twice: once for fixed
vertices, and once for non-fixed vertices on which mapping algorithms
are applied. However, for efficiency reasons (\eg, avoiding to compute
vertex distances that are equal to zero), it is preferable that
duplicate domains are avoided in the \texttt{domntab} array. This is
the case by nature with recursive bipartitioning, as the domains
associated with branches of the biparitioning tree are all distinct.

Making the distinction between fixed and non-fixed vertices, which is
relevant to mapping algorithms, is not in the scope of the
\texttt{Mapping} data structure, which only represents a global
state. This is why no data related to fixed vertices is explicitly
present in the mapping itself (it may be found, \eg, in the
\texttt{Kgraph} data structure).
However, for handling fixed vertices in an efficient way, the
semantics of the \texttt{Mapping} data structure is that all domains
that are associated with fixed vertices must be placed first in the
\texttt{domntab} array. The purpose of this separation is because,
when the imbalance of a mapping is computed, the loads of non-fixed
vertices that belong to some (partial) domain and of fixed vertices
that belong to domains that are subdomains of this domain have to be
aggregated. This aggregation procedure is made easier if both types of
domains are kept separate. For efficiency reasons, fixed domains
should appear only once in the fixed part of \texttt{domntab}.
\\

The \texttt{Mapping} structure is mainly used within the
\texttt{Kgraph} structure, which contains two instances of it: one for
the current mapping to be computed, and one for the old mapping, in
the case of remapping. The building of a \texttt{Kgraph} from another
one (\eg, a band graph or a coarsened graph) may lead to situations in
which some \texttt{Mapping} arrays may be re-used, and thus should not
be freed when the derived \texttt{Mapping} is freed. This is why the
\texttt{Mapping} structure contains flags to record whether its arrays
should be freed or not. These flags are the following:
\begin{itemize}
\item
\texttt{MAPPINGFREEDOMN}: set if the domain array has to be freed when
the mapping is freed. A common case for sharing the domain array is
when a coarser \texttt{Kgraph} is computed: the domain array of the
coarse old mapping can re-use that of the fine old mapping.
\item
\texttt{MAPPINGFREEPART}: set if the part array has to be freed when
the mapping is freed. A common case for sharing the part array is when
the user part array is kept as the part array for the initial
\texttt{Kgraph} current mapping structure.
\end{itemize}

The main fields of the \texttt{Mapping} data structure are the following:
\begin{itemize}
\item
\texttt{flagval}: set of flags that defines whether the
\texttt{parttax} and \texttt{domntab} have to be freed on exit.
\item
\texttt{grafptr}: pointer to the \texttt{Graph} associated with the
mapping, that gives access to the base value \texttt{grafptr->\lbt
baseval} and the number of source vertices \texttt{grafptr->\lbt
vertnbr}.
\item
\texttt{archptr}: pointer to the \texttt{Arch} associated with the
mapping, that is necessary to perform all distance computations on the
mapping.
\item
\texttt{parttax}: based array of \texttt{Anum}s, of size
\texttt{grafptr->\lbt vertnbr}, that provides the index of the target
domains onto which all graph vertices are currently mapped. Indices
are un-based.
\item
\texttt{domntab}: un-based array of \texttt{ArchDom}s, of size
\texttt{domnmax}, that stores the target domains to which source graph
vertices are indirectly associated through the \texttt{parttax} array.
\item
\texttt{domnnbr}: number of target domain slots currently used in
\texttt{domntab}. After a mapping is initialized, $1 \leq
\mbox{\texttt{domnnbr}} < \mbox{\texttt{domnmax}}$, because source
graph vertices must be associated to some domain, hence
\texttt{domntab} should at least contain one domain.
\item
\texttt{domnnbr}: number of target domain slots currently used in
\texttt{domntab}.
\item
\texttt{domnmax}: size of the \texttt{domntab} array.
\item
\texttt{mutedat}: when multi-threading is activated, allows to create
critical sections to update the mapping data in a thread-safe manner.
\end{itemize}
