Thank you for the amazing package! I was wondering if its possible to combine gradient caching with gradient accumulation and/or gradient checkpointing and if it is possible whether it even makes sense to do so. If you could provide an example of combining them in torch that would be a huge help!
Thank you for the amazing package! I was wondering if its possible to combine gradient caching with gradient accumulation and/or gradient checkpointing and if it is possible whether it even makes sense to do so. If you could provide an example of combining them in torch that would be a huge help!