Fourier Feature Pyramids for Physics-Informed Neural Networks

The gist The model beignet, which replaces random Fourier feature embedding with a learnable multi-resolution Fourier feature pyramid, achieves higher accuracy in compute-efficient manner by scaling parameter count instead of neural network architecture

How it works

1. beignet replaces the random Fourier feature embedding used by existing PINN models with a learnable multi-resolution Fourier feature pyramid

2. To query beignet at a continuous coordinate, Fourier interpolation is used at each level of the pyramid to return features at the input coordinate, and then these are decoded with a fully-connected neural network trunk

3. Spatial derivatives can be computed efficiently by using the chain rule to compose derivatives of the neural network computed with automatic differentiation with derivatives of the feature grid computed spectrally by the Fast Fourier transform (FFT)

4. beignet’s explicit control over spatial frequency content is particularly useful for PINNs because different PDEs have different spectral requirements

5. The finest grid resolution determines the maximum spatial frequency represented in the feature embedding, while the number of levels and channels per level controls the representational capacity within that band

Key Capabilities and Benefits

**: Spatial derivatives can be computed efficiently by using the chain rule to compose derivatives of the neural network computed with automatic differentiation with derivatives of the feature grid computed spectrally by the Fast Fourier transform (FFT) **

**: beignet can achieve higher accuracy in a compute-efficient manner by scaling the parameter count of this Fourier feature pyramid, instead of the less-efficient strategy of scaling the neural network architecture **

**: beignet can directly control the representation bandlimit, resulting in more stable optimization for difficult PDEs **

The model finds significantly more accurate solutions on PDE benchmarks using fewer parameters than state-of-the-art PINN methods. Furthermore, it minimizes residuals to near machine precision using Adam for the self-similar inviscid Burgers blowup problem, an accuracy regime previously attained only by using computationally expensive higher-order optimizers.

Architectural Components

**: The architecture relies on a learnable multi-resolution pyramid of periodic spatial feature grids, where each level stores a feature field at a prescribed grid resolution **

**: Evaluating the model at a point consists of two steps: first, using Fourier interpolation to query the input coordinate at each of the model’s levels, and then concatenating these resulting features before passing them through a decoder MLP **

**: The Fourier interpolation operation at the lth level can then be written as gθ(l) (x) = Re X k∈K(l) θb(l)[k] exp(2πi k · x), where K(l) is the set of signed FFT mode indices **

**: For time-dependent PDEs, the solution can vary sharply in space while evolving non-periodically in time, and the model divides the time axis into T bins to store T + 1 pyramids **

Performance on Benchmarks

beignet achieves substantially lower relative error across four JAX-PI benchmarks compared to JAX-PI and PirateNet architectures. Specifically, it reduces error by roughly an order of magnitude relative to PirateNet on Ginzburg–Landau. On the self-similar inviscid Burgers blowup problem, beignet drives the fixed-grid residuals to near machine precision using Adam, which is much more accurate than Adam-trained coordinate MLP baselines.

Design Choices and Ablations

**: The central design choice in beignet is not only the overall support of the Fourier features, but how capacity is distributed across resolutions, exposing two practical choices: the maximum resolution of the finest grid, which sets the representation’s frequency support, and the allocation of feature channels across levels **

**: The coarse-to-fine structure is a central part of the embedding because it allocates capacity across spatial bands and optimizes substantially better than stacking parametermatched feature grids at the finest resolution alone **

**: For KdV, adding very fine feature levels can hurt even when the parameter count is held fixed because high-frequency degrees of freedom are strongly amplified by the PDE operator, making optimization stiff **

**: The FFT-structured beignet residual path is substantially faster than the DFT/JVP evaluation of the same beignet model and PirateNet baseline in raw training-step time **

The structured residual path improves both the matched-NFE comparison and the feasible high-NFE training regime because it makes it practical to train with many more collocation points on a single GPU.

This work suggests that carefully designing the feature representation itself is a key ingredient for pushing PINNs into new high-accuracy regimes, opening the door to neural PDE solvers capable of resolving fine-scale structure, extreme multiscale dynamics, and challenging phenomena such as singularities. The authors manually verified all AI-generated code and results and substantially revised all AI-drafted text. The code is available at https://github.com/beignet-paper/beignet-release/. The authors would like to thank Matthieu Darcy for helpful discussions. This work was supported by the Amazon AI4Science Partnership Discovery Grant, Carver Mead New Adventures Fund, Ginsburg Prize, and Sloan Fellowship. The references include Ronen Basri et al. 2020 on Frequency bias in neural networks for input of non-uniform density and Charles L Fefferman 2006 on Existence and smoothness of the Navier-Stokes equation. The paper is available as arXiv:2605.24278v2 [cs.LG] 6 Oct 2026 The ISSN is 2835-8856. URL https://openreview.net/forum?id=KqRnsEMYLx The authors take full responsibility for the content of this paper. The authors are listed as Brandon Zhao, Yixuan Wang, Jonathan T. Barron, Katherine L. Bouman, Dor Verbin, and Pratul P. Srinivasan. The department affiliations are listed as Department of Computing and Mathematical Sciences at California Institute of Technology, Cahill Center for Astronomy and Astrophysics at California Institute of Technology, and Google DeepMind. The authors acknowledge the use of Generative AI tools to assist with implementing the method, running experiments, and drafting paper text. The authors are listed as Brandon Zhao (∗byzhao@caltech.edu). The references include Thomas Y. Hou and Pengfei Liu 2015 on Self-similar singularity of a 1D model for the 3D axisymmetric Euler equations and Yongji Wang et al. 2026 on High precision PINNs in unbounded domains: application to singularity formulation in PDEs. The references include Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar 2024 on Physics-informed neural operator for learning partial differential equations and Thomas Y. Hou and Pengfei Liu 2025 on KAN: Kolmogorov-Arnold networks. The references include Zongyi Li, Hongkai Zheng, Nikola Kovachki, David Jin, Haoxuan Chen, Burigede Liu, Kamyar Azizzadenesheli, and Anima Anandkumar 2024 on Physics-informed neural operator for learning partial differential equations. The references include Zongyi Li et al. 2021 on Fourier neural operator for parametric partial differential equations and Zongyi Li et al. 2025 on Fourier neural operator for parametric partial differential equations. The references include David B Lindell et al. 2024 on BACON: Band-limited coordinate networks for multiscale scene representation and Vincent Sitzmann et al. 2020 on Implicit neural representations with periodic activation functions. The references include Matthew Tancik et al. 2020 on Fourier features let networks learn high frequency functions in low dimensional domains. The references include Sifan Wang et al. 2024 on PirateNets: Physics-informed deep learning with residual adaptive networks. The references include Yixuan Wang et al. 2026 on On the expressiveness and spectral bias of KANs. The references include Yongji Wang et al. 2023b on Asymptotic selfsimilar blow-up profile for three-dimensional axisymmetric Euler equations using neural networks. The references include Yixuan Wang et al. 2026 on High precision PINNs in unbounded domains: application to singularity formulation in PDEs and Yongji Wang et al. 2025b on Discovery of unstable singularities.

1. textbfWeight-Controlled Spectral Allocation for PDE Structures: beignet architecture leverages a learnable multi-resolution Fourier feature pyramid to directly control the spatial bandwidth of the representation and the redundancy assigned to each frequency band. This allows the model to adapt its spectral capacity based on the PDE, as demonstrated by how it allocates features in Allen–Cahn versus KdV problems.

2. textbfEfficient Derivative Computation via FFT Composition: The system computes spatial derivatives by differentiating the feature fields in Fourier space and then propagating through the decoder by the chain rule, which preserves the bandlimited structure of each pyramid level while allowing the nonlinear decoder to mix features across scales. This method is shown to be substantially faster than pointwise evaluations, as indicated by Table 8.

3. textbfHigh-Precision Optimization via First-Order Methods: The architecture enables Adam [to] drive the PDE residual near machine precision with the same first-order optimizer, which suggests that representation design can reduce reliance on expensive second-order refinement for high-precision profile computations. This capability is demonstrated in the self-similar inviscid Burgers blowup problem, where it achieves a residual of 6.29 × 10−19 using Adam.

4. textbfreatment of Time as Non-Periodic Coordinate: For time-dependent problems, the model uses temporal pyramid blending to treat time as a non-periodic coordinate, allowing the representation to remain bandlimited in space while handling temporal evolution via linearly interpolat[ed] features. This is crucial for modeling non-periodic temporal dynamics.

5. textbfAdaptive Aliasing Mitigation: The inclusion of a stochastic grid shift during training helps mitigate aliasing issues by ensuring that the network does not learn high-frequency functions that satisfy the PDE residual at grid points while remaining inaccurate between them, as shown in Table 5.

We present an improved neural field architecture for solving partial differential equations (PDEs). Current physics-informed neural networks (PINNs) provide a flexible framework for solving PDEs, but they struggle to achieve highly accurate solutions and require computation that scales poorly with parameter count. Our model, which we call beignet (Bandlimited Embedding with Interpolated Grid Network), replaces the random Fourier feature embedding used by existing PINN models with a trainable multi-resolution Fourier feature pyramid. To query beignet at a continuous coordinate, we use Fourier interpolation at each level of the pyramid to return features at the input coordinate, and then decode this vector with a fully-connected neural network trunk. Our model provides multiple benefits: 1) Spatial derivatives can be computed efficiently by using the chain rule to compose derivatives of the neural network computed with automatic differentiation with derivatives of the feature grid computed spectrally by the Fast Fourier transform (FFT). 2) beignet can achieve higher accuracy in a compute-efficient manner by scaling the parameter count of this Fourier feature pyramid, instead of the less-efficient strategy of scaling the neural network architecture. 3) beignet can directly control the representation bandlimit, resulting in more stable optimization for difficult PDEs. We demonstrate that beignet finds significantly more accurate solutions on PDE benchmarks using fewer parameters than state-of-the-art PINN methods. We further evaluate beignet on the self-similar inviscid Burgers blowup problem and show that it can minimize residuals to near machine precision using Adam, an accuracy regime previously attained only by using computationally expensive higher-order optimizers.