Sparse matrix

check|unknown=|preview=Page using Template:Center with unknown parameter "_VALUE_"|ignoreblank=y| 1 | style }} <math>\left(\begin{smallmatrix} 11 & 22 & 0 & 0 & 0 & 0 & 0 \\ 0 & 33 & 44 & 0 & 0 & 0 & 0 \\ 0 & 0 & 55 & 66 & 77 & 0 & 0 \\ 0 & 0 & 0 & 0 & 0 & 88 & 0 \\ 0 & 0 & 0 & 0 & 0 & 0 & 99 \\ \end{smallmatrix}\right)</math> {{safesubst:#invoke:Check for unknown parameters|check|unknown=|preview=Page using Template:Center with unknown parameter "_VALUE_"|ignoreblank=y| 1 | style }}

Template:Resize {{safesubst:#invoke:Check for unknown parameters|check|unknown=|preview=Page using Template:Center with unknown parameter "_VALUE_"|ignoreblank=y| 1 | style }}

File:Finite element sparse matrix.png

A sparse matrix obtained when solving a finite element problem in two dimensions. The non-zero elements are shown in black.

In numerical analysis and scientific computing, a sparse matrix or sparse array is a matrix in which most of the elements are zero.<ref name="Yan Wu Liu Gao 2017 p. ">Template:Cite conference</ref> There is no strict definition regarding the proportion of zero-value elements for a matrix to qualify as sparse but a common criterion is that the number of non-zero elements is roughly equal to the number of rows or columns. By contrast, if most of the elements are non-zero, the matrix is considered dense.<ref name="Yan Wu Liu Gao 2017 p. "/> The number of zero-valued elements divided by the total number of elements (e.g., m × n for an m × n matrix) is sometimes referred to as the sparsity of the matrix.

Conceptually, sparsity corresponds to systems with few pairwise interactions. For example, consider a line of balls connected by springs from one to the next: this is a sparse system, as only adjacent balls are coupled. By contrast, if the same line of balls were to have springs connecting each ball to all other balls, the system would correspond to a dense matrix. The concept of sparsity is useful in combinatorics and application areas such as network theory and numerical analysis, which typically have a low density of significant data or connections. Large sparse matrices often appear in scientific or engineering applications when solving partial differential equations.

When storing and manipulating sparse matrices on a computer, it is beneficial and often necessary to use specialized algorithms and data structures that take advantage of the sparse structure of the matrix. Specialized computers have been made for sparse matrices,<ref>{{#invoke:citation/CS1|citation |CitationClass=web }}</ref> as they are common in the machine learning field.<ref>Template:Cite press release</ref> Operations using standard dense-matrix structures and algorithms are slow and inefficient when applied to large sparse matrices as processing and memory are wasted on the zeros. Sparse data is by nature more easily compressed and thus requires significantly less storage. Some very large sparse matrices are infeasible to manipulate using standard dense-matrix algorithms.

Special cases

Banded

Template:Main article An important special type of sparse matrices is band matrix, defined as follows. The lower bandwidth of a matrix Template:Math is the smallest number Template:Math such that the entry Template:Math vanishes whenever Template:Math. Similarly, the upper bandwidth is the smallest number Template:Math such that Template:Math whenever Template:Math Template:Harv. For example, a tridiagonal matrix has lower bandwidth Template:Math and upper bandwidth Template:Math. As another example, the following sparse matrix has lower and upper bandwidth both equal to 3. Notice that zeros are represented with dots for clarity. <math display="block">\begin{bmatrix}

 X   &   X   &   X   & \cdot & \cdot & \cdot & \cdot & \\
 X   &   X   & \cdot &   X   &   X   & \cdot & \cdot & \\
 X   & \cdot &   X   & \cdot &   X   & \cdot & \cdot & \\

\cdot & X & \cdot & X & \cdot & X & \cdot & \\ \cdot & X & X & \cdot & X & X & X & \\ \cdot & \cdot & \cdot & X & X & X & \cdot & \\ \cdot & \cdot & \cdot & \cdot & X & \cdot & X & \\ \end{bmatrix}</math>

Matrices with reasonably small upper and lower bandwidth are known as band matrices and often lend themselves to simpler algorithms than general sparse matrices; or one can sometimes apply dense matrix algorithms and gain efficiency simply by looping over a reduced number of indices.

By rearranging the rows and columns of a matrix Template:Math it may be possible to obtain a matrix Template:Math with a lower bandwidth. A number of algorithms are designed for bandwidth minimization.

Diagonal

A very efficient structure for an extreme case of band matrices, the diagonal matrix, is to store just the entries in the main diagonal as a one-dimensional array, so a diagonal Template:Math matrix requires only Template:Math entries.

Symmetric

A symmetric sparse matrix arises as the adjacency matrix of an undirected graph; it can be stored efficiently as an adjacency list.

Block diagonal

A block-diagonal matrix consists of sub-matrices along its diagonal blocks. A block-diagonal matrix Template:Math has the form <math display="block">\mathbf{A} = \begin{bmatrix}

 \mathbf{A}_1 & 0             & \cdots & 0            \\
 0            & \mathbf{A}_2  & \cdots & 0            \\
 \vdots       & \vdots        & \ddots & \vdots       \\
 0            & 0             & \cdots & \mathbf{A}_n

\end{bmatrix},</math>

where Template:Math is a square matrix for all Template:Math.

Use

Reducing fill-in

The fill-in of a matrix are those entries that change from an initial zero to a non-zero value during the execution of an algorithm. To reduce the memory requirements and the number of arithmetic operations used during an algorithm, it is useful to minimize the fill-in by switching rows and columns in the matrix. The symbolic Cholesky decomposition can be used to calculate the worst possible fill-in before doing the actual Cholesky decomposition.

There are other methods than the Cholesky decomposition in use. Orthogonalization methods (such as QR factorization) are common, for example, when solving problems by least squares methods. While the theoretical fill-in is still the same, in practical terms the "false non-zeros" can be different for different methods. And symbolic versions of those algorithms can be used in the same manner as the symbolic Cholesky to compute worst case fill-in.

Solving sparse matrix equations

Both iterative and direct methods exist for sparse matrix solving.

Iterative methods, such as conjugate gradient method and GMRES utilize fast computations of matrix-vector products <math>A x_i</math>, where matrix <math>A</math> is sparse. The use of preconditioners can significantly accelerate convergence of such iterative methods.

Template:Anchor Storage

A matrix is typically stored as a two-dimensional array. Each entry in the array represents an element Template:Math of the matrix and is accessed by the two indices Template:Math and Template:Math. Conventionally, Template:Math is the row index, numbered from top to bottom, and Template:Math is the column index, numbered from left to right. For an Template:Math matrix, the amount of memory required to store the matrix in this format is proportional to Template:Math (disregarding the fact that the dimensions of the matrix also need to be stored).

In the case of a sparse matrix, substantial memory requirement reductions can be realized by storing only the non-zero entries. Depending on the number and distribution of the non-zero entries, different data structures can be used and yield huge savings in memory when compared to the basic approach. The trade-off is that accessing the individual elements becomes more complex and additional structures are needed to be able to recover the original matrix unambiguously.

Formats can be divided into two groups:

• Those that support efficient modification, such as DOK (Dictionary of keys), LIL (List of lists), or COO (Coordinate list). These are typically used to construct the matrices.
• Those that support efficient access and matrix operations, such as CSR (Compressed Sparse Row) or CSC (Compressed Sparse Column).

Dictionary of keys (DOK)

DOK consists of a dictionary that maps Template:Math-pairs to the value of the elements. Elements that are missing from the dictionary are taken to be zero. The format is good for incrementally constructing a sparse matrix in random order, but poor for iterating over non-zero values in lexicographical order. One typically constructs a matrix in this format and then converts to another more efficient format for processing.<ref>See scipy.sparse.dok_matrix</ref>

List of lists (LIL)

LIL stores one list per row, with each entry containing the column index and the value. Typically, these entries are kept sorted by column index for faster lookup. This is another format good for incremental matrix construction.<ref>See scipy.sparse.lil_matrix</ref>

Coordinate list (COO)

COO stores a list of Template:Math tuples. Ideally, the entries are sorted first by row index and then by column index, to improve random access times. This is another format that is good for incremental matrix construction.<ref>See scipy.sparse.coo_matrix</ref>

Compressed sparse row (CSR, CRS or Yale format)

The compressed sparse row (CSR) or compressed row storage (CRS) or Yale format represents a matrix Template:Math by three (one-dimensional) arrays, that respectively contain nonzero values, the extents of rows, and column indices. It is similar to COO, but compresses the row indices, hence the name. This format allows fast row access and matrix-vector multiplications (Template:Math). The CSR format has been in use since at least the mid-1960s, with the first complete description appearing in 1967.<ref>Template:Cite conference</ref>

The CSR format stores a sparse Template:Math matrix Template:Math in row form using three (one-dimensional) arrays Template:Math. Let Template:Math denote the number of nonzero entries in Template:Math. (Note that zero-based indices shall be used here.)

• The arrays Template:Math and Template:Math are of length Template:Math, and contain the non-zero values and the column indices of those values respectively
• Template:Math contains the column in which the corresponding entry Template:Math is located.
• The array Template:Math is of length Template:Math and encodes the index in Template:Math and Template:Math where the given row starts. This is equivalent to Template:Math encoding the total number of nonzeros above row Template:Math. The last element is Template:Math , i.e., the fictitious index in Template:Math immediately after the last valid index Template:Math.<ref name=Saad03>Template:Harvnb</ref>

For example, the matrix <math display="block">\begin{pmatrix} 5 & 0 & 0 & 0 \\ 0 & 8 & 0 & 0 \\ 0 & 0 & 3 & 0 \\ 0 & 6 & 0 & 0 \\ \end{pmatrix}</math> is a Template:Math matrix with 4 nonzero elements, hence

V         = [ 5 8 3 6 ]
COL_INDEX = [ 0 1 2 1 ]
ROW_INDEX = [ 0 1 2 3 4 ] 

assuming a zero-indexed language.

To extract a row, we first define:

row_start = ROW_INDEX[row]
row_end   = ROW_INDEX[row + 1]

Then we take slices from V and COL_INDEX starting at row_start and ending at row_end.

To extract the row 1 (the second row) of this matrix we set row_start=1 and row_end=2. Then we make the slices V[1:2] = [8] and COL_INDEX[1:2] = [1]. We now know that in row 1 we have one element at column 1 with value 8.

In this case the CSR representation contains 13 entries, compared to 16 in the original matrix. The CSR format saves on memory only when Template:Math.

Another example, the matrix <math display="block">\begin{pmatrix} 10 & 20 & 0 & 0 & 0 & 0 \\

0 & 30 &  0 & 40 &  0 &  0 \\
0 &  0 & 50 & 60 & 70 &  0 \\
0 &  0 &  0 &  0 &  0 & 80 \\

\end{pmatrix}</math> is a Template:Math matrix (24 entries) with 8 nonzero elements, so

V         = [ 10 20 30 40 50 60 70 80 ]
COL_INDEX = [  0  1  1  3  2  3  4  5 ]   
ROW_INDEX = [  0  2  4  7  8 ]

The whole is stored as 21 entries: 8 in Template:Math, 8 in Template:Math, and 5 in Template:Math.

• Template:Math splits the array Template:Math into rows: (10, 20) (30, 40) (50, 60, 70) (80), indicating the index of Template:Math (and Template:Math) where each row starts and ends;
• Template:Math aligns values in columns: (10, 20, ...) (0, 30, 0, 40, ...)(0, 0, 50, 60, 70, 0) (0, 0, 0, 0, 0, 80).

Note that in this format, the first value of Template:Math is always zero and the last is always Template:Math, so they are in some sense redundant (although in programming languages where the array length needs to be explicitly stored, Template:Math would not be redundant). Nonetheless, this does avoid the need to handle an exceptional case when computing the length of each row, as it guarantees the formula Template:Math works for any row Template:Math. Moreover, the memory cost of this redundant storage is likely insignificant for a sufficiently large matrix.

The (old and new) Yale sparse matrix formats are instances of the CSR scheme. The old Yale format works exactly as described above, with three arrays; the new format combines Template:Math and Template:Math into a single array and handles the diagonal of the matrix separately.<ref>Template:Citation</ref>

For logical adjacency matrices, the data array can be omitted, as the existence of an entry in the row array is sufficient to model a binary adjacency relation.

It is likely known as the Yale format because it was proposed in the 1977 Yale Sparse Matrix Package report from Department of Computer Science at Yale University.<ref>{{#invoke:citation/CS1|citation |CitationClass=web }}</ref>

Compressed sparse column (CSC or CCS)

CSC is similar to CSR except that values are read first by column, a row index is stored for each value, and column pointers are stored. For example, CSC is Template:Math, where Template:Math is an array of the (top-to-bottom, then left-to-right) non-zero values of the matrix; Template:Math is the row indices corresponding to the values; and, Template:Math is the list of Template:Math indexes where each column starts. The name is based on the fact that column index information is compressed relative to the COO format. One typically uses another format (LIL, DOK, COO) for construction. This format is efficient for arithmetic operations, column slicing, and matrix-vector products. This is the traditional format for specifying a sparse matrix in MATLAB (via the sparse function).

Software

Many software libraries support sparse matrices, and provide solvers for sparse matrix equations. The following are open-source:

• PETSc, a large C library, containing many different matrix solvers for a variety of matrix storage formats.
• Trilinos, a large C++ library, with sub-libraries dedicated to the storage of dense and sparse matrices and solution of corresponding linear systems.
• Eigen3 is a C++ library that contains several sparse matrix solvers. However, none of them are parallelized.
• MUMPS (MUltifrontal Massively Parallel sparse direct Solver), written in Fortran90, is a frontal solver.
• deal.II, a finite element library that also has a sub-library for sparse linear systems and their solution.
• DUNE, another finite element library that also has a sub-library for sparse linear systems and their solution.
• Armadillo provides a user-friendly C++ wrapper for BLAS and LAPACK.
• SciPy provides support for several sparse matrix formats, linear algebra, and solvers.
• ALGLIB is a C++ and C# library with sparse linear algebra support
• ARPACK Fortran 77 library for sparse matrix diagonalization and manipulation, using the Arnoldi algorithm
• SLEPc Library for solution of large scale linear systems and sparse matrices
• scikit-learn, a Python library for machine learning, provides support for sparse matrices and solvers
• SparseArrays is a Julia standard library.
• PSBLAS, software toolkit to solve sparse linear systems supporting multiple formats also on GPU.

History

The term sparse matrix was possibly coined by Harry Markowitz who initiated some pioneering work but then left the field.<ref>Oral history interview with Harry M. Markowitz, pp. 9, 10.</ref>

References

• Template:Cite book
• Template:Cite book
• Template:Cite book (This book, by a professor at the State University of New York at Stony Book, was the first book exclusively dedicated to Sparse Matrices. Graduate courses using this as a textbook were offered at that University in the early 1980s).
• {{#invoke:citation/CS1|citation

|CitationClass=web }}

• Template:Cite book
• Template:Cite journal Also NOAA Technical Memorandum NOS NGS-4, National Geodetic Survey, Rockville, MD. Referencing Template:Harvnb.
• Template:Cite book (Open Access)

Further reading

• Template:Cite journal
• Template:Cite journal
• Sparse Matrix Algorithms Research at the Texas A&M University.
• SuiteSparse Matrix Collection
• SMALL project A EU-funded project on sparse models, algorithms and dictionary learning for large-scale data.
• Template:Cite book
• Template:Cite book
• Template:Cite book