Critical Patch-Aware Sparse Prompting with Decoupled Training for Continual Learning on the Edge

Wonseon Lim Jaesung Lee* Dae-Won Kim*
Chung-Ang University CVPR 2026
*Corresponding authors

Task-aware token sparsity and decoupled optimization for efficient prompt-based continual learning on resource-constrained edge hardware.

PDF arXiv Code CVPR Poster
Overview of CPS-Prompt with critical patch sampling and decoupled prompt and classifier training

CPS-Prompt selects task-relevant patches from the query stage and separates sparse prompt learning from full-patch classifier alignment.

1.6x

average improvement in peak memory, training time, and energy efficiency over CODA-Prompt

< 2%

average accuracy gap to state-of-the-art C-Prompt

3

benchmarks: CIFAR-100, ImageNet-R, and CUB-200

Jetson

edge-device measurements on NVIDIA Jetson Orin Nano

Abstract

Continual learning on edge devices requires high accuracy and training-time efficiency under strict memory and computation constraints. Prompt-based continual learning is parameter-efficient, but prior methods often prioritize accuracy or inference-time speed while overlooking the cost of on-device training.

CPS-Prompt addresses this bottleneck with Critical Patch Sampling, which uses task-aware signals from the frozen query pass to keep informative patches, and Decoupled Prompt and Classifier Training, which reduces backpropagation overhead while aligning sparse training with full-patch inference. Across public benchmarks and real edge hardware, CPS-Prompt improves memory, time, and energy efficiency while maintaining competitive accuracy.

Method

Critical Patch Sampling computes patch importance from class-token attention and value norms in the final query transformer block. A temperature-scaled multinomial sampler retains a compact subset of task-relevant patches before the prompt-injected forward pass.

Decoupled Prompt and Classifier Training divides each task into a sparse prompt training phase and a full-patch classifier alignment phase. This reduces stored activations during prompt learning and mitigates the representation mismatch introduced by sparse inputs.

CPS-Prompt framework diagram

Results

CPS-Prompt is designed for the accuracy-efficiency trade-off that matters during continual training on edge devices.

Accuracy and Efficiency

CPS-Prompt remains competitive with CODA-Prompt and stays within about 2% average accuracy of C-Prompt, while substantially reducing peak memory, wall-clock training time, and energy consumption.

Accuracy, forgetting, memory, training time, and energy comparison across CIFAR-100, ImageNet-R, and CUB-200

Task-Aware Sparsity

Compared with task-agnostic token reduction methods such as ToMe and PatchDropout, CPS-Prompt better preserves accuracy as memory reduction increases because patch selection follows task-relevant query signals.

Accuracy versus memory reduction across benchmarks

Training-Time Scaling

As patch reduction increases, CPS-Prompt lowers training time across benchmarks while maintaining a stronger accuracy-efficiency balance than task-agnostic token reduction baselines.

Training time versus memory reduction across benchmarks

Ablation Signal

CPS improves over random patch dropping at the same memory usage, and DPCT recovers accuracy while lowering training time through classifier-only alignment.

Ablation table for CPS and DPCT on ImageNet-R

BibTeX

@inproceedings{lim2026cpsprompt,
  title     = {Critical Patch-Aware Sparse Prompting with Decoupled Training for Continual Learning on the Edge},
  author    = {Lim, Wonseon and Lee, Jaesung and Kim, Dae-Won},
  booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  year      = {2026},
  url       = {https://arxiv.org/abs/2604.07399}
}