Multidimensional scaling

File:RecentVotes.svg

An example of classical multidimensional scaling applied to voting patterns in the United States House of Representatives. Each blue dot represents one Democratic member of the House, and each red dot one Republican.

Multidimensional scaling (MDS) is a means of visualizing the level of similarity of individual cases of a data set. MDS is used to translate distances between each pair of <math display="inline"> n </math> objects in a set into a configuration of <math display="inline"> n </math> points mapped into an abstract Cartesian space.<ref name="MS_history">Template:Cite journal</ref>

More technically, MDS refers to a set of related ordination techniques used in information visualization, in particular to display the information contained in a distance matrix. It is a form of non-linear dimensionality reduction.

Given a distance matrix with the distances between each pair of objects in a set, and a chosen number of dimensions, N, an MDS algorithm places each object into N-dimensional space (a lower-dimensional representation) such that the between-object distances are preserved as well as possible. For N = 1, 2, and 3, the resulting points can be visualized on a scatter plot.<ref name="borg">Template:Cite book</ref>

Core theoretical contributions to MDS were made by James O. Ramsay of McGill University, who is also regarded as the founder of functional data analysis.<ref name="jsto_ACon">Template:Cite journal</ref>

Types

MDS algorithms fall into a taxonomy, depending on the meaning of the input matrix:

Classical multidimensional scaling

It is also known as Principal Coordinates Analysis (PCoA), Torgerson Scaling or Torgerson–Gower scaling. It takes an input matrix giving dissimilarities between pairs of items and outputs a coordinate matrix whose configuration minimizes a loss function called strain,<ref name="borg"/> which is given by <math display=block>\text{Strain}_D(x_1,x_2,...,x_n)=\Biggl(\frac{ \sum_{i,j} \bigl( b_{ij} - x_i^T x_j \bigr)^2}{\sum_{i,j}b_{ij}^2} \Biggr)^{1/2},</math> where <math>x_{i}</math> denote vectors in N-dimensional space, <math>x_i^T x_j </math> denotes the scalar product between <math>x_{i}</math> and <math>x_{j}</math>, and <math>b_{ij}</math> are the elements of the matrix <math>B</math> defined on step 2 of the following algorithm, which are computed from the distances.

Steps of a Classical MDS algorithm:

Classical MDS uses the fact that the coordinate matrix <math>X</math> can be derived by eigenvalue decomposition from <math display="inline">B=XX'</math>. And the matrix <math display="inline">B</math> can be computed from proximity matrix <math display="inline">D</math> by using double centering.<ref>Wickelmaier, Florian. "An introduction to MDS." Sound Quality Research Unit, Aalborg University, Denmark (2003): 46</ref>

1. Set up the squared proximity matrix <math display="inline">D^{(2)}=[d_{ij}^2]</math>
2. Apply double centering: <math display="inline">B=-\frac{1}{2}CD^{(2)}C</math> using the centering matrix <math display="inline">C=I-\frac{1}{n}J_n</math>, where <math display="inline">n</math> is the number of objects, <math display="inline">I</math> is the <math display="inline">n \times n</math> identity matrix, and <math display="inline">J_{n}</math> is an <math display="inline">n\times n</math> matrix of all ones.
3. Determine the <math display="inline">m</math> largest eigenvalues <math display="inline">\lambda_1,\lambda_2,...,\lambda_m</math> and corresponding eigenvectors <math display="inline">e_1,e_2,...,e_m</math> of <math display="inline">B</math> (where <math display="inline">m</math> is the number of dimensions desired for the output).
4. Now, <math display="inline">X=E_m\Lambda_m^{1/2}</math> , where <math display="inline">E_m</math> is the matrix of <math display="inline">m</math> eigenvectors and <math display="inline">\Lambda_m</math> is the diagonal matrix of <math display="inline">m</math> eigenvalues of <math display="inline">B</math>.

Classical MDS assumes metric distances. So this is not applicable for direct dissimilarity ratings.

Metric multidimensional scaling (mMDS)

It is a superset of classical MDS that generalizes the optimization procedure to a variety of loss functions and input matrices of known distances with weights and so on. A useful loss function in this context is called stress, which is often minimized using a procedure called stress majorization. Metric MDS minimizes the cost function called “stress” which is a residual sum of squares:

<math>\text{Stress}_D(x_1,x_2,...,x_n)=\sqrt{\sum_{i\ne j=1,...,n}\bigl(d_{ij}-\|x_i-x_j\|\bigr)^2}.</math>

Metric scaling uses a power transformation with a user-controlled exponent <math display="inline">p</math>: <math display="inline">d_{ij}^p</math> and <math display="inline">-d_{ij}^{2p}</math> for distance. In classical scaling <math display="inline">p=1.</math> Non-metric scaling is defined by the use of isotonic regression to nonparametrically estimate a transformation of the dissimilarities.

Non-metric multidimensional scaling (NMDS)

In contrast to metric MDS, non-metric MDS finds both a non-parametric monotonic relationship between the dissimilarities in the item-item matrix and the Euclidean distances between items, and the location of each item in the low-dimensional space.

Let <math>d_{ij}</math> be the dissimilarity between points <math>i, j</math>. Let <math>\hat d_{ij} = \| x_i - x_j\|</math> be the Euclidean distance between embedded points <math>x_i, x_j</math>.

Now, for each choice of the embedded points <math>x_i</math> and is a monotonically increasing function <math>f</math>, define the "stress" function:

<math>S(x_1, ..., x_n; f)=\sqrt{\frac{\sum_{i<j}\bigl(f(d_{ij})-\hat d_{ij}\bigr)^2}{\sum_{i<j} \hat d_{ij}^2}}.</math>

The factor of <math>\sum_{i<j} \hat d_{ij}^2</math> in the denominator is necessary to prevent a "collapse". Suppose we define instead <math>S=\sqrt{\sum_{i<j}\bigl(f(d_{ij})-\hat d_{ij})^2}</math>, then it can be trivially minimized by setting <math>f = 0</math>, then collapse every point to the same point.

A few variants of this cost function exist. MDS programs automatically minimize stress in order to obtain the MDS solution.

The core of a non-metric MDS algorithm is a twofold optimization process. First the optimal monotonic transformation of the proximities has to be found. Secondly, the points of a configuration have to be optimally arranged, so that their distances match the scaled proximities as closely as possible.

NMDS needs to optimize two objectives simultaneously. This is usually done iteratively:

1. Initialize <math>x_i</math> randomly, e. g. by sampling from a normal distribution.
2. Do until a stopping criterion (for example, <math>S < \epsilon</math>)
   1. Solve for <math>f = \arg\min_f S(x_1, ..., x_n ; f)</math> by isotonic regression.
   2. Solve for <math>x_1, ..., x_n = \arg\min_{x_1, ..., x_n} S(x_1, ..., x_n ; f)</math> by gradient descent or other methods.
3. Return <math>x_i</math> and <math>f</math>

Louis Guttman's smallest space analysis (SSA) is an example of a non-metric MDS procedure.

Generalized multidimensional scaling (GMD)

{{#invoke:Labelled list hatnote|labelledList|Main article|Main articles|Main page|Main pages}} An extension of metric multidimensional scaling, in which the target space is an arbitrary smooth non-Euclidean space. In cases where the dissimilarities are distances on a surface and the target space is another surface, GMDS allows finding the minimum-distortion embedding of one surface into another.<ref name="bron">Template:Cite journal</ref>

Super multidimensional scaling (SMDS)

An extension of MDS, known as Super MDS, incorporates both distance and angle information for improved source localization. Unlike traditional MDS, which uses only distance measurements, Super MDS processes both distance and angle-of-arrival (AOA) data algebraically (without iteration) to achieve better accuracy.<ref>Template:Cite conference</ref>

The method proceeds in the following steps:

1. Construct the Reduced Edge Gram Kernel: For a network of <math>N</math> sources in an <math>\eta</math>-dimensional space, define the edge vectors as <math>v_{i} = x_{m} - x_{n}</math>. The dissimilarity is given by <math>k_{i,j} = \langle v_i, v_j \rangle</math>. Assemble these into the full kernel <math>K = VV^T</math>, and then form the reduced kernel using the <math>N-1</math> independent vectors: <math>\bar{K} = [V]_{(N-1)\times\eta}\ [V]_{(N-1)\times\eta}^T</math>,
2. Eigen-Decomposition: Compute the eigen-decomposition of <math>\bar{K}</math>,
3. Estimate Edge Vectors: Recover the edge vectors as <math> \hat{V} = \Bigl( U_{M \times \eta}\, \Lambda^{\odot \frac{1}{2}}_{\eta \times \eta} \Bigr)^T </math>,
4. Procrustes Alignment: Retrieve <math>\hat{V}</math> from <math>V</math> via Procrustes Transformation,
5. Compute Coordinates: Solve the following linear equations to compute the coordinate estimates <math>\begin{pmatrix}

1 \vline \mathbf{0}_{1 \times N-1} \\ \hline \mathbf{[C]}_{N-1 \times N} \end{pmatrix} \cdot \begin{pmatrix}\mathbf{x}_{1} \\ \hline[\mathbf{X}]_{N-1 \times \eta} \end{pmatrix}=\begin{pmatrix} \mathbf{x}_{1} \\ \hline[\mathbf{V}]_{N-1 \times \eta} \end{pmatrix}, </math>

This concise approach reduces the need for multiple anchors and enhances localization precision by leveraging angle constraints.

Details

The data to be analyzed is a collection of <math>M</math> objects (colors, faces, stocks, . . .) on which a distance function is defined,

<math>d_{i,j} :=</math> distance between <math>i</math>-th and <math>j</math>-th objects.

These distances are the entries of the dissimilarity matrix

<math> D :=

\begin{pmatrix} d_{1,1} & d_{1,2} & \cdots & d_{1,M} \\ d_{2,1} & d_{2,2} & \cdots & d_{2,M} \\ \vdots & \vdots & & \vdots \\ d_{M,1} & d_{M,2} & \cdots & d_{M,M} \end{pmatrix}. </math>

The goal of MDS is, given <math>D</math>, to find <math>M</math> vectors <math>x_1,\ldots,x_M \in \mathbb{R}^N</math> such that

<math>\|x_i - x_j\| \approx d_{i,j}</math> for all <math>i,j\in {1,\dots,M}</math>,

where <math>\|\cdot\|</math> is a vector norm. In classical MDS, this norm is the Euclidean distance, but, in a broader sense, it may be a metric or arbitrary distance function.<ref name="Kruskal">Kruskal, J. B., and Wish, M. (1978), Multidimensional Scaling, Sage University Paper series on Quantitative Application in the Social Sciences, 07-011. Beverly Hills and London: Sage Publications.</ref> For example, when dealing with mixed-type data that contain numerical as well as categorical descriptors, Gower's distance is a common alternative.Template:Cn

In other words, MDS attempts to find a mapping from the <math>M</math> objects into <math>\mathbb{R}^N</math> such that distances are preserved. If the dimension <math>N</math> is chosen to be 2 or 3, we may plot the vectors <math>x_i</math> to obtain a visualization of the similarities between the <math>M</math> objects. Note that the vectors <math>x_i</math> are not unique: With the Euclidean distance, they may be arbitrarily translated, rotated, and reflected, since these transformations do not change the pairwise distances <math>\|x_i - x_j\|</math>.

(Note: The symbol <math>\mathbb{R}</math> indicates the set of real numbers, and the notation <math>\mathbb{R}^N</math> refers to the Cartesian product of <math>N</math> copies of <math>\mathbb{R}</math>, which is an <math>N</math>-dimensional vector space over the field of the real numbers.)

There are various approaches to determining the vectors <math>x_i</math>. Usually, MDS is formulated as an optimization problem, where <math>(x_1,\ldots,x_M)</math> is found as a minimizer of some cost function, for example,

<math> \underset{x_1,\ldots,x_M}{\mathrm{argmin}} \sum_{i<j} ( \|x_i - x_j\| - d_{i,j} )^2. \, </math>

A solution may then be found by numerical optimization techniques. For some particularly chosen cost functions, minimizers can be stated analytically in terms of matrix eigendecompositions.<ref name="borg" />

Procedure

There are several steps in conducting MDS research:

1. Formulating the problem – What variables do you want to compare? How many variables do you want to compare? What purpose is the study to be used for?
2. Obtaining input data – For example, :- Respondents are asked a series of questions. For each product pair, they are asked to rate similarity (usually on a 7-point Likert scale from very similar to very dissimilar). The first question could be for Coke/Pepsi for example, the next for Coke/Hires rootbeer, the next for Pepsi/Dr Pepper, the next for Dr Pepper/Hires rootbeer, etc. The number of questions is a function of the number of brands and can be calculated as <math>Q = N (N - 1) / 2</math> where Q is the number of questions and N is the number of brands. This approach is referred to as the “Perception data : direct approach”. There are two other approaches. There is the “Perception data : derived approach” in which products are decomposed into attributes that are rated on a semantic differential scale. The other is the “Preference data approach” in which respondents are asked their preference rather than similarity.
3. Running the MDS statistical program – Software for running the procedure is available in many statistical software packages. Often there is a choice between Metric MDS (which deals with interval or ratio level data), and Nonmetric MDS<ref>Template:Cite journal</ref> (which deals with ordinal data).
4. Decide number of dimensions – The researcher must decide on the number of dimensions they want the computer to create. Interpretability of the MDS solution is often important, and lower dimensional solutions will typically be easier to interpret and visualize. However, dimension selection is also an issue of balancing underfitting and overfitting. Lower dimensional solutions may underfit by leaving out important dimensions of the dissimilarity data. Higher dimensional solutions may overfit to noise in the dissimilarity measurements. Model selection tools like AIC, BIC, Bayes factors, or cross-validation can thus be useful to select the dimensionality that balances underfitting and overfitting.
5. Mapping the results and defining the dimensions – The statistical program (or a related module) will map the results. The map will plot each product (usually in two-dimensional space). The proximity of products to each other indicate either how similar they are or how preferred they are, depending on which approach was used. How the dimensions of the embedding actually correspond to dimensions of system behavior, however, are not necessarily obvious. Here, a subjective judgment about the correspondence can be made (see perceptual mapping).
6. Test the results for reliability and validity – Compute R-squared to determine what proportion of variance of the scaled data can be accounted for by the MDS procedure. An R-square of 0.6 is considered the minimum acceptable level. Template:Citation needed An R-square of 0.8 is considered good for metric scaling and .9 is considered good for non-metric scaling. Other possible tests are Kruskal’s Stress, split data tests, data stability tests (i.e., eliminating one brand), and test-retest reliability.
7. Report the results comprehensively – Along with the mapping, at least distance measure (e.g., Sorenson index, Jaccard index) and reliability (e.g., stress value) should be given. It is also very advisable to give the algorithm (e.g., Kruskal, Mather), which is often defined by the program used (sometimes replacing the algorithm report), if you have given a start configuration or had a random choice, the number of runs, the assessment of dimensionality, the Monte Carlo method results, the number of iterations, the assessment of stability, and the proportional variance of each axis (r-square).

Implementations

• ELKI includes two MDS implementations.
• MATLAB includes two MDS implementations (for classical (cmdscale) and non-classical (mdscale) MDS respectively).
• The R programming language offers several MDS implementations, e.g. base cmdscale function, packages smacof<ref>Template:Cite journal</ref> (mMDS and nMDS), and vegan (weighted MDS).
• scikit-learn contains function sklearn.manifold.MDS.

See also

• Data clustering
• t-distributed stochastic neighbor embedding
• Factor analysis
• Discriminant analysis
• Dimensionality reduction
• Distance geometry
• Cayley–Menger determinant
• Sammon mapping
• Iconography of correlations

Bibliography

• Template:Cite book
• Template:Cite book
• Template:Cite journal
• Template:Cite book
• Template:Cite book
• Template:Cite book