How to manage your Google Colab state and storage

Google Colab is a popular cloud-based platform for running Python code, conducting data analysis, and training machine learning models. It offers free access to computing resources, but limited storage space can sometimes become a challenge. If you encounter a "No space left on the device" error, here are some strategies to manage your Google Colab state and storage effectively.

| If you are interested in how to monitor your disk take a look at this article .

Purchase Google Colab Pro

If you find yourself frequently running out of storage in Google Colab, investing in Google Colab Pro is an excellent option. Google Colab Pro more than doubles the amount of disk space available to you on both GPU and CPU environments. Colab Pro provides a host of additional benefits, including faster GPU access, longer run times, and priority access to resources. Compared to other cloud offerrings Google Colab Pro has very competive pricing but does come with some limitations.

Free up unused space

Another approach to manage your Google Colab state and storage is to free up unused space within your environment. Often, temporary files or data from previous runs can accumulate and consume valuable disk space. You can perform the following steps to clean up your Colab environment:

1. Delete Unnecessary Files: Identify and delete any files that are no longer needed. To find what's taking up space you can use !du -sh {path}. Then use the !rm command to remove files directly from the Colab environment.
2. Clear Output and Variables: Clear the output of code cells and variables that are no longer required. This can be done using the Runtime -> Restart runtime option in the Colab menu.
3. Restart Kernel: Restarting the kernel will clear all variables and free up memory. You can do this by selecting Runtime -> Restart runtime in the Colab menu.

By regularly cleaning up your Colab environment, you can optimize your storage space and ensure that only necessary files and data are retained.

Offload Files

If upgrading to Google Colab Pro is not an option or you still require more storage space, you can offload your files to external storage services. Google Colab provides seamless integration with popular cloud storage providers like Google Drive and Google Cloud Storage.

To offload files to Google Drive, you can mount your Google Drive in your Colab environment using the following code snippet:

from google.colab import drive
drive.mount('/content/drive')

This will allow you to access your Google Drive files within the Colab environment. You can then move your large datasets, model checkpoints, or any other files that are consuming significant disk space to your Google Drive. Remember to manage your storage on Google Drive to prevent it from getting full as well.

Then unmount it when done to free up space.

drive.flush_and_unmount()

Connecting a google drive and mounting it in Colab does not increase the size of your local disk even if you have upgraded your Google Drive/One plan to the 2TB plan or above.

Use a Google Colab Alternative

Other notebook services may have bigger disk space and more options for configuration. Check out our latest article on the best Colab alternatives.

One notable one (I may be biased) is Brev which gives you full control over your compute and storage options allowing you to configure a disk drive up to 64TB.

Conclusion

Managing storage space in Google Colab can be a challenge, especially when working with large datasets or complex machine learning models. However, there are several effective ways to optimize your Colab environment's storage capacity. By purchasing Google Colab Pro, freeing up unused space, offloading files to external storage services, and considering alternative platforms, you can ensure that your environment has sufficient storage space.