Tensor Cookbook: Mastering Tensors through Diagrams

Tensor networks provide an effective framework for addressing challenges in manipulating high-dimensional data by using graphical notation to encode tensor contractions and decompositions. This manuscript offers a self-contained guide to tensor networks, demonstrating how this diagrammatic approach simplifies proofs of classical identities, rank bounds, and gradient formulas that would otherwise require laborious index manipulation.

Tensor Networks Basics

The paper introduces Tensor Networks (TNs) as graphs where vertices represent tensors and edges represent contracted indices <ref:2605.16610#pg2>. Tensors are represented by shapes with legs, where the number of legs corresponds to the order of the tensor, and contractions are encoded as shared edges <ref:2605.16610#pg9>. A key feature is that un-contracted legs correspond to free indices, and their number indicates the order of a tensor <ref:2605.16610#pg13>. The paper details how inner products, outer products, trace operations, and Frobenius norms can be generalized to higher-order tensors using this formalism <ref:2605.16610#pg14>.

Operations on Tensors

The manuscript covers fundamental tensor operations through diagrammatic proofs <ref:2605.16610#pg3> and formal definitions. These include permutation and reshaping operations like matricization, where the grouping of indices is represented by merging corresponding legs together <ref:2605.16610#pg17>. Tensor products are also discussed, such as the mode-n product, which generalizes matrix multiplication along a specific mode <ref:2605.16610#pg3> and vector products that reduce the tensor order by one <ref:2605.16610#pg3>. The Kronecker product is shown to be equivalent to a reshaping of outer products, and several useful properties are listed, such as the mixed product property (A ⊗ B)(C ⊗ D) = AC ⊗ BD <ref:2605.16610#pg20>.

Tensor Decompositions

The paper presents three principal decomposition models for high-dimensional tensors <ref:2605.16610#pg4> and discusses their tensor network representations. The CANDECOMP/PARAFAC (CP) decomposition expresses a tensor as a sum of rank-one terms, which is represented by a diagram involving the third-order copy tensor <ref:2605.16610#pg26>. The Tucker decomposition generalizes this via a compressed core tensor and factor matrices, and its multi-linear rank is defined as the smallest tuple controlling expressivity <ref:2605.16610#pg31>. The Tensor Train (TT) decomposition provides a chain-structured factorization whose parameter count scales linearly with tensor order, and its rank can be viewed as a hyperparameter controlling expressivity <ref:2605.16610#pg32>.

Computing Gradients of Tensor Networks

The manuscript develops a graphical calculus for the differentiation of tensor network functions <ref:2605.16610#pg41>. Theorem 19 establishes that the Jacobian of a tensor network with respect to a core tensor is obtained by removing that core from the diagram, yielding transparent proofs of classical derivative identities like ∂(Ax)/∂A = x ◦ I <ref:2605.16610#pg39>. The chain rule can be extended to expressions involving tensors, and applications include gradient-based optimization of the CP decomposition and backpropagation through weight matrices parameterized in Matrix Product Operator (MPO) format <ref:2605.16610#pg43>.

Probability Distributions and Random Tensor Networks

Tensor networks are used to model high-dimensional probability distributions efficiently <ref:2605.16610#pg54>. Joint probability distributions can be encoded as tensors, and marginal probabilities can be expressed through simple diagrammatic operations by contracting a vector of ones with corresponding legs <ref:2605.16610#pg44>. The Tensor Train format provides an efficient, structured parameterization of high-dimensional distributions, such as Born Machine-style representations <ref:2605.16610#pg46>. Furthermore, the expected values of random tensor networks can be computed using diagrammatic reasoning and Isserlis’ theorem to derive closed-form expressions for quantities like E[∥AB∥ 2 F] <ref:2605.16610#pg47>.

Conclusion

The manuscript concludes by showing how manipulating tensor network expressions can help derive non-trivial results in an intuitive manner <ref:2605.16610#pg54>. The graphical language of tensor networks is presented as a clear and precise mathematical framework that brings together many classical results into a single, unified structure <ref:2605.16610#pg55>. This approach leads to representations that are both compact and conceptually transparent when working with high-dimensional tensors <ref:2605.16610#pg55>.

The gist

Tensor networks provide an effective framework for addressing challenges in manipulating high-dimensional data by using graphical notation to encode tensor contractions and decompositions.

How it works

Tensors are represented by shapes with legs, where the number of legs corresponds to the order of the tensor, and contractions are encoded as shared edges <ref:2605.16610#pg2>.

The CP decomposition expresses a tensor as a sum of rank-one terms, which is represented by a diagram involving the third-order copy tensor <ref:2605.16610#pg26>.

The Tensor Train (TT) decomposition provides a chain-structured factorization whose parameter count scales linearly with tensor order <ref:2605.16610#pg32>.

The Jacobian of a tensor network with respect to a core tensor is obtained by removing that core from the diagram <ref:2605.16610#pg39>.

**"The probabilistic computations of Chapter 5 show that non-trivial results, including the mean of the Wishart distribution (Proposition 24), Isserlis’ theorem for Gaussian tensors (Theorem 27), and the expected norm of the product of Gaussian random matrices (Proposition 26) admit concise derivations using tensor network notation" <ref:2605.16610#pg54>.

1. Tensor network based neural networks can be parameterized using Matrix Product Operator format to significantly reduce memory usage for dense weight matrices in fully connected layers, as described in Section 3.4. This allows AI systems to handle much larger models that would otherwise be computationally infeasible due to exponential parameter growth.

2. Gradient computation for tensor network functions is simplified by Theorem 19, which states that the Jacobian of a tensor network with respect to a core tensor is obtained simply by removing that core from the diagram. This enables efficient backpropagation through large neural networks without laborious index manipulation.

3. Probability distributions can be efficiently modeled using Tensor Train decomposition to parameterize high-dimensional joint probability distributions as low-rank tensors, such as Born Machine representations. This allows for the compact representation of complex statistical models in machine learning contexts.

4. Expected values of random tensor networks, such as E(ATA) for Gaussian matrices, can be computed efficiently using tensor network notation and propositions like Proposition 28. This provides a direct method to calculate complex statistical properties of random weights without intricate index summations.

5. Loss functions involving tensors, such as the Frobenius norm loss L = A - T 2F, have their gradients easily derived using chain rule extensions based on Theorem 19 and Khatri-Rao products. This provides a direct path for optimizing tensor network models in machine learning tasks like CP decomposition or neural network compression.

6. Tensor decompositions like Tucker and Tensor Train allow for the computation of multi-linear rank, which is given by the rank of the matricizations (Theorem 14). This provides a measure of the expressivity and complexity of high-order tensor representations, guiding model compression strategies.

7. Matrix factorization methods like SVD can be directly represented in tensor network diagrams, allowing for efficient computation of ranks and singular values as shown in Proposition 15. This is crucial for algorithms that rely on low-rank approximations of high-dimensional data or weight matrices.

High-dimensional data arise naturally in many areas of science and engineering, including machine learning, signal processing, computational physics, and statistics. Such data are often represented as tensors, multi-dimensional generalizations of matrices. While tensors provide a natural representation for multi-modal structure, their direct manipulation quickly becomes challenging as the order grows: the number of parameters increases exponentially, and algebraic expressions involving many indices become difficult to interpret and implement. Tensor networks (TNs) provide an effective framework for addressing these challenges. Originally introduced by Penrose and developed extensively in quantum physics, the graphical language of tensor networks encodes contractions as edges in a graph, reducing notational overhead and revealing structural properties obscured by index notation. Despite the central role of high-dimensional tensors in modern machine learning and numerical analysis, tensor network diagrams remain underutilized outside quantum computing, partly due to the lack of a self-contained mathematical reference accessible to a broad technical audience. This manuscript provides a self-contained guide to tensor networks and their use in tensor algebra. We present the main operations on tensors, contractions, products, and reshaping through, graphical notation, and show how classical tensor decompositions and related computations are naturally expressed in this framework. We also illustrate how tensor networks simplify the derivation of gradients and the manipulation of high-dimensional probability distributions. Throughout, we show that the diagrammatic approach yields genuinely shorter and more transparent proofs of classical identities, rank bounds, and gradient formulas that would otherwise require laborious index manipulation.