Replicate is a model hosting and discovery platform. It has a pay-per-use plan that allows you to be billed down to the second for fine-tuning and inference tasks without having to set up your own infrastructure.
As of May 2023, Replicate's prices are as follows
For similar GPUs on Google Cloud, the prices are as follows
Since Replicate is pay-per-use, some calls may incur a cold start cost increasing latency. If doing a lot of predictions or fine-tuning, it may make sense to run your models on always-on dedicated infrastructure like Google Cloud, AWS, or Lambda Labs. Typically running your models on cloud VMs can be a bit of a headache, as you have to deal with VPCs, firewalls, machine images, instance types, quotas, key pairs, and different kinds of volumes, file systems, operating systems, driver versions, not to mention instance availability. Luckily, we’ll be using two tools to simplify all of this: Cog and Brev. Cog simplifies running production-ready containers for machine learning. Brev finds and provisions AI-ready instances across various cloud providers.
Create a Brev.dev account with payment or a cloud connect
Find the model and Cog Container on Replicate
In this guide, we will be using joehoover’s musicgen model as an example. Go to the model Replicate page and select API → Docker. Copy and save the run command.
Configure the Brev Instance
Create a new Brev instance. Add an install script, select “I don’t have a bash script” then “Show script”. In the input field enter the command from above as root by prepending
sudo. This will run your model on port 5000 ready for inference. Read more about cog deployments in the cog docs. Finally, click “Done”.
Select GPU and Deploy
Select the GPU you want to use, name the instance, then create.
Expose your model
Click “Share a Service” then enter the port your model is running on.
By default, the model API is private so only your Brev user can access it in a browser you are logged in on. To consume the API in your applications you have two choices: API key or public. To create an API key, select “API Key” then “Generate Key”. To make your service totally public, click edit access and toggle on public.
Replicate is a fantastic tool to easily deploy and discover AI/ML models in a pay-per-use manner. However, with a lot of use, it can be an expensive option. We looked at using Brev and Cog to deploy Replicate models on dedicated GPUs on various clouds.