Spatial Intelligence
PDF

Notation

5 min read

Symbols are listed in chapter order, with the meanings used in the referenced sections.

Symbol Description Reference
ℰ\mathcal{E} Physical or simulated environment. Sec. 2.2.2
𝒮\mathcal{S} AI system or agent. Sec. 2.2.2
𝒯,t∈𝒯\mathcal{T},\;t\in\mathcal{T} Task space and a task in that space. Sec. 2.2.2
ℛ𝒮\mathcal{R}_{\mathcal{S}} Internal spatial representation space of the system. Sec. 2.2.2
Φ𝒮\Phi_{\mathcal{S}} Mapping from the environment to the internal spatial representation space. Sec. 2.2.2
ℱ𝒮\mathcal{F}_{\mathcal{S}} Set of operators for perception, reasoning, prediction, and action. Sec. 2.2.2
𝒪,𝒜\mathcal{O},\;\mathcal{A} Observation space and action space, respectively. Sec. 2.2.2
fperc𝒮f^{\mathcal{S}}_{\mathrm{perc}} Operator that maps observations to spatial representations. Sec. 2.2.2
freason𝒮f^{\mathcal{S}}_{\mathrm{reason}} Reasoning operator inferring relations and constraints in the current representation. Sec. 2.2.2
fpred𝒮f^{\mathcal{S}}_{\mathrm{pred}} Prediction operator mapping a representation and an action to a future representation. Sec. 2.2.2
fact𝒮f^{\mathcal{S}}_{\mathrm{act}} Action operator mapping a spatial representation to an action. Sec. 2.2.2
π𝒮\pi_{\mathcal{S}} Task-conditioned policy for selecting actions. Sec. 2.2.2
ℛt𝒮\mathcal{R}^{\mathcal{S}}_{t} Internal spatial representation state at time tt. Sec. 2.2.2
at𝒮,ot+1a^{\mathcal{S}}_{t},\;o_{t+1} Selected action and subsequent observation. Sec. 2.2.2
Ψ𝒮\Psi_{\mathcal{S}} Update function for the internal spatial representation. Sec. 2.2.2
Ω\Omega Image grid or domain of pixel coordinates. Sec. 4.1.1
I,VI,\;V Image and video, respectively. Sec. 4.1.1
S,ℒS,\;\mathcal{L} Segmentation map and its set of labels. Sec. 4.1.2
D⁢(u,v)D(u,v) Depth value at pixel (u,v)(u,v). Sec. 4.1.3
𝐍⁢(u,v)\mathbf{N}(u,v) Unit surface normal at pixel (u,v)(u,v). Sec. 4.1.4
𝐅⁢(u,v)\mathbf{F}(u,v) Optical flow giving horizontal and vertical pixel displacement. Sec. 4.1.6
𝒦,Hk\mathcal{K},\;H_{k} Set of keypoint locations and the heatmap for keypoint kk. Sec. 4.1.7
P={pi}P=\{p_{i}\} Point cloud with 3D coordinates and optional attributes. Sec. 4.2.3
M=(V,F)M=(V,F) Triangle mesh with vertex set VV and face set FF. Sec. 4.2.4
f⁢(𝐱)f(\mathbf{x}) Implicit scalar field whose level set defines a surface; a signed distance function in the SDF case. Sec. 4.2.5
FΘF_{\Theta} Neural radiance field mapping position and viewing direction to color and volume density. Sec. 4.2.6
𝐱,𝐝\mathbf{x},\;\mathbf{d} 3D query position and viewing direction in a radiance field. Sec. 4.2.6
σ,𝐜\sigma,\;\mathbf{c} Volume density and emitted RGB color in a radiance field. Sec. 4.2.6
𝝁,𝚺\bm{\mu},\;\bm{\Sigma} Center and covariance matrix of a 3D Gaussian primitive. Sec. 4.2.7
αi,ci\alpha_{i},\;c_{i} Opacity and color of the ii-th Gaussian primitive. Sec. 4.2.7
Δ⁢𝐱⁢(t)\Delta\mathbf{x}(t) Time-dependent displacement from a reference 3D position. Sec. 4.3.4
𝒢⁢(t)\mathcal{G}(t) Time-dependent collection of Gaussian primitives in a dynamic scene. Sec. 4.3.5
bi,ci,sib_{i},\;c_{i},\;s_{i} Bounding box, class label, and confidence score of detection ii. Sec. 6.1.1
fθ,θf_{\theta},\;\theta Parameterized prediction model and its learned parameters. Sec. 6.1.3
𝕀⁢(⋅)\mathbb{I}(\cdot) Indicator function, equal to one when its condition holds and zero otherwise. Sec. 6.1.3
Z,f,B,dZ,\;f,\;B,\;d Stereo depth, focal length, camera baseline, and disparity, with Z=f⁢B/dZ=fB/d. Sec. 6.2.1
KK Camera intrinsic matrix. Sec. 6.2.3
R,𝐭R,\;\mathbf{t} Rotation and translation mapping world coordinates into the camera frame. Sec. 6.2.3
𝐓t\mathbf{T}_{t} Camera pose at time tt, represented by a rigid transformation. Sec. 6.2.4
Δ⁢𝐓i→j\Delta\mathbf{T}_{i\to j} Relative camera transformation from frame ii to frame jj. Sec. 6.2.4
𝒢\mathcal{G} Scene graph of subject–predicate–object triplets. Sec. 6.3.2
𝐯⁢(𝐱)\mathbf{v}(\mathbf{x}) Scene flow: 3D displacement of the surface point observed at pixel 𝐱\mathbf{x}. Sec. 6.4.1
𝒯k\mathcal{T}_{k} Time-indexed trajectory of tracked object kk. Sec. 6.4.2
SO⁢(3),SE⁢(3)\mathrm{SO}(3),\;\mathrm{SE}(3) Groups of 3D rotations and rigid transformations, respectively. Sec. 7.1.1
𝐅\mathbf{F} Fundamental matrix relating corresponding image points in two views. Sec. 7.1.1
𝐪,𝐉⁢(𝐪)\mathbf{q},\;\mathbf{J}(\mathbf{q}) Joint configuration and the Jacobian mapping joint velocities to task-space velocities. Sec. 7.1.1
𝒞free\mathcal{C}_{\mathrm{free}} Feasible configuration space used in reachability and motion planning. Sec. 7.1.1
𝒲\mathcal{W} Set of contact wrenches used to test grasp force closure. Sec. 7.1.1
𝐬t,𝐮t\mathbf{s}_{t},\;\mathbf{u}_{t} Physical system state and applied action in the dynamics model. Sec. 7.1.2
M⁢(𝐪)M(\mathbf{q}) Configuration-dependent mass matrix. Sec. 7.1.2
G=(V,E,w)G=(V,E,w) Planning graph with vertices, edges, and edge weights. Sec. 7.1.3
do⁡(X=x)\operatorname{do}(X=x) Intervention that sets variable XX to xx in a causal model. Sec. 7.1.6
z⋆z^{\star} Optimal solution returned by a differentiable optimization layer. Sec. 7.2.4
πθ\pi_{\theta} Learned policy for selecting actions. Sec. 7.2.5
rt,γr_{t},\;\gamma Reward at time tt and the discount factor. Sec. 7.2.5
Q∗⁢(s,a)Q^{*}(s,a) Maximum expected discounted return from taking action aa in state ss. Sec. 7.2.5
RtR_{t} Discounted return from time tt onward. Sec. 7.2.5
Vϕ,AϕV_{\phi},\;A_{\phi} State-value function and advantage function. Sec. 7.2.5
at:t+Ha_{t:t+H} Action sequence predicted over horizon HH by a vision-language-action policy. Sec. 7.3.2
τ=(t1,…,tL)\tau=(t_{1},\ldots,t_{L}) Sequence of intermediate reasoning steps in a chain of thought. Sec. 7.3.3
ℳt\mathcal{M}_{t} External spatial memory after the update at time tt. Sec. 7.3.6
𝒳,𝐱\mathcal{X},\;\mathbf{x} Space of tokenized text prompts and an input prompt. Sec. 8.1.1
𝒴,𝐲\mathcal{Y},\;\mathbf{y} Pixel-space image domain and an image in that domain. Sec. 8.1.1
𝒵,𝐳\mathcal{Z},\;\mathbf{z} Latent representation space and a latent code. Sec. 8.1.1
E,DE,\;D Image encoder and decoder of the latent autoencoder. Sec. 8.1.1
ϕ,ℰ\phi,\;\mathcal{E} Text encoder and its conditioning embedding space. Sec. 8.1.1
GθG_{\theta} Conditional generator or sampler. Sec. 8.1.1
pθ(⋅∣⋅)p_{\theta}(\cdot\mid\cdot) Learned conditional distribution over generated outputs or latent codes. Sec. 8.1.1
ϵθ\epsilon_{\theta} Noise predictor used in diffusion generation. Sec. 8.1.1
𝐳t\mathbf{z}_{t} Latent code at diffusion or flow time tt. Sec. 8.1.1
vθv_{\theta} Learned velocity field used in flow matching. Sec. 8.1.1
𝐜,𝒞\mathbf{c},\;\mathcal{C} Structural conditioning signal and its input space. Sec. 8.1.2
κ,𝒞′\kappa,\;\mathcal{C}^{\prime} Condition encoder and the space of encoded conditions. Sec. 8.1.2
ℳ\mathcal{M} Mask identifying the region to be edited. Sec. 8.1.3
ℛ,𝐫\mathcal{R},\;\mathbf{r} Space of 3D representations and a 3D asset in that space. Sec. 8.2.1
π,πv\pi,\;\pi_{v} Camera pose used for rendering, with vv indexing a view. Sec. 8.2.1
Render\mathrm{Render} Renderer mapping a 3D representation and a camera pose to an image. Sec. 8.2.1
ℒSDS\mathcal{L}_{\mathrm{SDS}} Score-distillation objective for optimizing a representation with a frozen diffusion prior. Sec. 8.2.1
ℒrec\mathcal{L}_{\mathrm{rec}} Loss measuring agreement between a reconstruction and its observations. Sec. 8.2.3
λ,Ω\lambda,\;\Omega Weight and regularizer used in scene reconstruction. Sec. 8.2.3
Λ\Lambda Operator lifting a panorama or a set of panoramic views into a 3D representation. Sec. 8.2.4
EθE_{\theta} Editing operator mapping a 3D representation and a condition to an edited representation. Sec. 8.2.5
𝐦1:T\mathbf{m}^{1:T} Sequence of body poses describing a motion. Sec. 8.4.1
𝐫1:T\mathbf{r}^{1:T} Time-indexed sequence of 3D representations for a dynamic scene. Sec. 8.5
WθW_{\theta} Learned world model predicting the next state from state, action, and observation. Sec. 8.6
S⁢(t),A⁢(t),O⁢(t)S(t),\;A(t),\;O(t) World state, action, and observation, respectively. Sec. 8.6
Eϕ,PθE_{\phi},\;P_{\theta} Observation encoder and latent-state predictor in a representation world model. Sec. 8.6.1
sg⁡(⋅)\operatorname{sg}(\cdot) Stop-gradient operator used in representation learning. Sec. 8.6.1
M⁢(t)M(t) Latent mental state of another agent, including beliefs, desires, and intentions. Sec. 8.6.3
stphy,stments_{t}^{\mathrm{phy}},\;s_{t}^{\mathrm{ment}} Physical and mental components of a joint world state. Sec. 8.6.3
ρ,ℒ\rho,\;\mathcal{L} Symbolic scene program and its language. Sec. 8.7.1
R⁢(ρ)R(\rho) Execution or rendering of symbolic scene program ρ\rho. Sec. 8.7.1
MtM_{t} Semantic environment map maintained during warehouse localization and mapping. Sec. 9.5.4