Batch size is a proxy for negative set entropy per update. SimCLR relies on batch entropy for stability; MoCo supplies entropy through a memory state instead.
This single sentence encapsulates the entire divergence between two paradigms of self-supervised learning.
Both systems depend on one critical object: The diversity of negatives seen per parameter update. Conceptually: The effective negative set richness.
Collapse pressure increases when:
Negatives become too similar
Negatives become too few in variety
Negatives become too correlated with positives
Stability requires: Sufficiently high diversity in the negative distribution.
SimCLR has only one source of negatives: the current batch.
Batch size directly controls negative diversity.
Pathology | Mechanism | Consequence |
|---|---|---|
Low contrast entropy | Few negatives per update | Model sees only narrow slice of representation space |
Fast self-reinforcement | Limited negatives → overfitting | Overfitting to augmentation artifacts, transient geometry |
Correlated gradient field | Few independent repulsive directions | Gradient field becomes low-rank, noisy but uninformative |
System Behavior: A stochastic process with insufficient spanning directions in its constraint forces.
Collapse occurs when gradient directions fail to span representation space and all samples are pushed into a shared attractor region.
The system cannot maintain geometric separation pressure.
MoCo introduces two mechanisms:
Mechanism | Function |
|---|---|
EMA encoder (slow state) | Temporal smoothing, prevents rapid geometric drift |
Queue (historical negatives) | Memory of past representations |
Negative diversity now comes from Time, not Batch Size.
Instant entropy: Current batch (Limited).
Temporal entropy: Queue of past embeddings (Large, decorrelated across time).
MoCo decouples negative diversity from batch size. The queue effectively creates a "virtual batch" spanning many past iterations.
State evolution depends on current parameters + current batch only.
Negative entropy per step ∝ Batch Size.
Small batch → low entropy → collapse-prone.
State evolution depends on current parameters (fast), EMA parameters (slow), and Queue (historical).
Negative entropy per step ∝ (Batch + Temporal Memory).
Batch size is NOT the bottleneck; memory dominates entropy supply.
Define C = covariance of contrastive gradient directions induced by negatives.
Eigenvalues of C measure how many independent "separation directions" exist.
rank(C) = number of effective independent repulsion axes.
Collapse condition (spectral form): Collapse occurs when C becomes effectively low-rank. Stability condition: Effective rank of C must exceed the intrinsic collapse mode dimension.
rank(C) ≤ B − 1
Stability requires: B ≥ r_collapse + 1.
C_total = C_batch + C_queue + C_EMA
r_eff ≈ r_batch + r_temporal
Even small B can produce large r_eff.
Continuous-time abstraction: dθ/dt = f(θ, batch)
Linearization: d(δz)/dt = J · δz (where J is the Jacobian of embedding dynamics).
Collapse occurs when the eigenvalue along the global contraction mode (v_c) becomes dominant or insufficiently damped.
J = J_contrast (Repulsion) + J_regularization (Smoothing/Damping)
System | Collapse Eigenvalue | Key Feature |
|---|---|---|
SimCLR |
| Depends only on batch size; no memory term |
MoCo |
| Includes temporal rank expansion (σ_c) and EMA damping (γ) |
As Q → ∞, embeddings become a continuous distribution over past model states.
Finite queue: Contrastive dynamics depend on finite-rank matrix C.
Infinite queue: C → K (Integral Operator).
In the infinite queue limit, collapse is no longer prevented by increasing rank, but by spectral spreading of the memory kernel. The mechanism shifts from "Rank sufficiency" to "Spectral dispersion stability."
Projecting onto collapse eigenfunction: da/dt = λ_c · a Mean-field stability: E[a(t)] = a(0) · exp((A − γ)t) Where A = (λ̄batch + σ_c).
γ > λ̄batch + σ_c
EMA acts as a spectral shift operator. If stable, the collapse dies exponentially at rate γ − A.
Queue stores Q samples along trajectory with mixing time τ_mix. Correlation decay: Corr(z_t, z_{t+k}) ∝ exp(−k/τ_mix)
Q_crit(γ) ∝ exp(c · γ)
Reasoning: EMA introduces exponential temporal compression (horizon 1/γ). Queue must span exponentially many correlated states to cover one EMA "effective memory unit."
τ_EMA (1/γ): EMA memory horizon.
τ_Q (τ_mix · log Q): Queue coverage/diversity scale.
τ_mix: Intrinsic decorrelation rate.
Region | Condition | Behavior |
|---|---|---|
(I) Collapse |
| EMA dominates; embeddings collapse |
(II) Stable |
| Goldilocks Zone: Structured semantic geometry |
(III) Fragmentation |
| Excessive historical entropy; loss of coherent clustering |
SimCLR requires large batches because it must achieve high-rank contrastive geometry in a single optimization step, whereas MoCo distributes rank accumulation across time via EMA and the memory queue, decoupling stability from batch size.
Algorithmic to Physical: Training as a dynamical system with entropy reservoirs.
Discrete to Continuous: Infinite queue reveals operator-theoretic structure.
Static to Phase-based: Stability exists on a manifold.
Empirical to Spectral: All phenomena reduce to eigenvalue structure.
End of Framework Consolidation. Zero Data Loss Confirmed.