Exam NCA-GENM Topic 1 Question 213 Discussion

Gradient checkpointing (also known as activation recomputation) trades compute for memory. It avoids storing all intermediate activations during the forward pass, instead recomputing them during the backward pass. This significantly reduces memory footprint, allowing you to train larger models or use larger batch sizes. Increasing batch size increases memory consumption. Decreasing gradient accumulation steps might help slightly but is less effective than gradient checkpointing. Switching to a smaller model or disabling mixed precision would degrade model performance.