Disentangling Attention in Deep Operator Learning: A Controlled Study of Data-Driven and Physics-Informed Architectures

A controlled study on a variant of DeepONet revealed the significant impact of the attention mechanism on the accuracy of solving partial differential equations (PDEs). The researchers tested five different DeepONet variants with two architecture modes: data-driven and physical information-based, and evaluated them on benchmark problems such as source-driven transient one-dimensional nonlinear diffusion-reaction equations, transient one-dimensional viscous Burgers equations, and two-dimensional Poisson heat conduction problems. The results showed that models using a perceptron segmentation and cross-attention configuration reduced the average relative L_2 error by 2.4 to 28.0 times in all benchmark combinations, with the best configuration reducing it by 3.5 to 32.3 times; while the branch-of-attention mechanism using only dot-product fusion performed inconsistently. Although叠加 cross-attention improved all six scenarios, the improvement was minimal, global pre-mixing did not provide consistent benefits, and increasing the depth of cross-attention led to a significant increase in cost and a decrease in benefits. Overall, query-dependent cross-attention was proven to improve accuracy…