MongoDB 8.0 Performance is 36% higher, but there is a catch…

TLDR: If your app is performance critical, think twice, thrice before upgrading to MongoDB 7.0 and 8.0.

Here is why…

One day I was reading about something and landed up on a blog post shared by the CTO of the MongoDB. The post surprised me with a claim that the MongoDB 8.0 is their the most secure and performant version ever. He claimed a whoping 36% to 50% reduction in Query Time. I don’t know know why but the post was not satisfying my mind. So I looked for few comparisons post, benchmarking results, tweak and tune settings post with comparing results like “before” and “after”.

This search led me to an interesting thread posted on MongoDB’s official community forum. The thread titled Performance Drop After Upgrade 6.0.10 > 7.0.1 was more interesting , practical and insightful than the many one-sided benchmark results. Plus, it was all contributed by community members (users) of MongoDB (community & Atlas both).

One of the user starts the above thread and posts some details about his/her odd findings, and asks for confirmation-cum-clarification.

The user’s query got a reply from MongoDB Employee, asking a few questions and a guess for root cause.

The user then confirms, and also shares a finding that the “if downgraded to 6.0 again, restores the same performance”.

And there goes a list of other users claiming the exact same behaviour that emphasises the performance issue with Mongo DB 7.0. I am not posting those many screenshots to cut-short the blog length, but you can always refer to the above thread link and checkout yourself.

MongoDB 8.0 claims to be faster on reads by 36% is nowhere mentioned from which version they compared the results and did the math. The claim is actually by comparing the MongoDB 8.0 against MongoDB 7.0. I’ll try to prove later on..

The Irony

The MongoDB 7.0 was indeed a bad performer (until 7.0.14-rc was released). Imagine the performance of your existing MongoDB 6.0 gives a query result in fraction of seconds and when used same query and dataset on MongoDB 7.0 it gives result in ~7s (from one of the user’s finding). Now, if MongoDB 8.0 is compared against 7.0, and the results are 36-55 % faster, the real performance of MongoDB 8.0 as compared to 6.0 will be actually still smaller.

What was the reason behind the performance drop?

Since I do not have knowledge and publicly available data, I’ll share my analysis based on my findings and will try to put resource references wherever possible.

Remember the reply of MongoDB employee above, talking about slot based query engine (also known as SBE, henceforth) was actually the culprit.
One another user mentions two JIRA ticket IDs, SERVER-94735 and SERVER-62150.

The JIRA ticket SERVER-62150 is a bug, that mentions the affected versions were 5.1 to 7.0.4 and it was fixed in 8.0.0-rc0

The JIRA ticket SERVER-94735 is a task that aims to set the default value of internalQueryFrameworkControl to forceClassicEngine . This indicates there was surely a problem with SBE (as guessed the MongoDB employee from the beginning).

Another user found interestingly a new thing, he mentioned that on MongoDB 7.0 the WiredTiger Tickets were dropped to ~8 tickets, whereas in MongoDB 6.0 it was ~128.

So then he mentions the fix as follows:

We managed to resolve the problem by setting  storageEngineConcurrentWriteTransactions and  storageEngineConcurrentReadTransactions to a fixed value of 128, just as in MongoDB 6. Afterwards the performance also returned to MongoDB 6 levels!

Is the performance issue fixed then?

Based on the JIRA and descriptions, it seems to be fixed in MongoDB 8.0. Based on the above tuning settings also, it seems to work fine.

But also, since the performance drop is significantly larger and if your application is sensitive to performance, ideally its not recommended to upgrade to MongoDB 7.0 and MongoDB 8.0. However, you can give it a try to upgrade to MongoDB 8.0 and observe the performance. If there is a performance dip seen for your workload, you should revert back to 6.0.

If you are okay to experiment, you can try to tune using the above settings of storageEngineConcurrentWriteTransactions and storageEngineConcurrentReadTransactions to 128. See if it helps, you can try the same in production too, rollback otherwise.

If you are not sure, my suggestion is to keep using MongoDB 6.x until further clarifications and more benchmarks prove the fixes.