Reduce the cost of OpenSearch Serverless in development

Reduce the cost of OpenSearch Serverless in development

We released a WordPress plugin that utilizes Amazon Bedrock Knowledge Bases a few weeks ago:

RAG Chat for Amazon Bedrock – WordPress plugin | WordPress.org

Amazon Bedrock Knowledge Bases has a few options for its backend vector store such as OpenSearch Serverless, Amazon Aurora PostgreSQL-Compatible Edition and Pinecone. We started using OpenSearch Serverless for our demo site last month:

RAG Chat for Amazon Bedrock – WordPress + RAG Demo Site

OpenSearch Serverless was costing too much due to redundancy

It was working fine, but I got a budget alert email from AWS. I opened the console, and it showed that OpenSearch Serverless had cost $100+ for only 10 days or so. I had only created 3 small collections for demo and development purposes.

I checked out the following pricing page:

Amazon OpenSearch Service – Pricing

Then, I found the following:

Additionally, OpenSearch Serverless also offers a dev-test option, where you can launch a collection without redundant standby nodes. This deployment mode further cuts the cost in half, with 0.5 OCU for indexing and 0.5 OCU for search.

The redundancy is enabled by default. If I had created the collections through the management console, I would have probably noticed the setting, but I used CDK. The management console looks like the following:

How to disable the redundancy

Unfortunately, this setting can’t be changed once you create an index. The only option (at least currently) is to remove the index, and create a new one. When it’s disabled, the collection page looks like the following:

The tricky part is that “At this time, capacity settings only apply at the account level. You can’t configure per-collection capacity limits.” See the following document for more details:

Managing capacity limits for Amazon OpenSearch Serverless – Amazon OpenSearch Service

So, if you have five collections for development purposes, and the redundancy setting is enabled for one of them, the minimum OCUs required would still be 2 (1 for indexing and 1 for search). If the redundancy is disabled for all of the collections, the minimum OCUs would go down to 1 (0.5 each for indexing and search).

In my case, I had three collections, all of which had redundancy enabled. I recreated them with redundancy disabled, and the number of OCUs went down to 0.5 each as shown below:

Conclusion

In OpenSearch Serverless, redundancy is enabled by default. If your collections are for development or demo purposes, you can disable it to cut the cost in half.

The number of OCUs is determined based on the redundancy setting and the workload, and the capacity (OCUs) seems to be shared per account and region. So, if you have many collections with redundancy disabled, but only one enabled, the minimum number of OCUs is still 2 in the region, while it’s 1 if redundancy is disabled for all the collections.

we are hiring

優秀な技術者と一緒に、好きな場所で働きませんか

株式会社もばらぶでは、優秀で意欲に溢れる方を常に求めています。働く場所は自由、働く時間も柔軟に選択可能です。

現在、以下の職種を募集中です。ご興味のある方は、リンク先をご参照下さい。

コメントを残す