SurealDb performance/benchmark transparency

[Snazzie]

Many have been asking for benchmarks since surrealDbs announcement, but have always been met by unwillingness to provide benchmark data. Quick search on reddit shows a few users complain about abysmal performance, worse than Sqlite. Seeing SurrealDb claiming its trusted by teams at big name companies raises my eyebrows.

[nickchomey]

It's a newish tool with a very broad and ambitious goal, so maybe cut them some slack? They only have so many hands.

In a recent live Q&A on discord, Tobie confirmed to me that v1 isn't really meant to be "production-ready", but just stable. V2, which will hopefully be released in the next few months, will come with performance improvements, benchmarks and benchmarking tools.

Have some patience!

[solmanter]

Hi surreal team. I am really happy to new announcement and features. does new features includes benchmarking tools?

[Snazzie]

is it just me or does performance feel like a day 1 consideration when building a groundbreaking database 🤔 it isnt some enterprise application where speed is second class.

[nickchomey]

A counterpoint to that is this blog post https://motherduck.com/blog/perf-is-not-enough

And the general adage for programming "make it work, make it right, make it fast"

Surrealdb is enormously ambitious and just had to be made to work at all to begin with. Then they've seemingly had to refactor a lot to make it "right" (including a rewrite in rust).

Now, it seems, that it mostly works (though there's some key missing features, glaring bugs, as well as, I think, scope creep), so making it fast is now becoming more of a focus.

Still, I think making a benchmarking suite should have been a top priority early on, just so you could track and analyze how changes affected perf. And certainly before releasing a "production-ready" v2, which also didn't have adequate (or even existent) SDKs. Those were big errors.

Anyway, it is what it is. The team is obviously working very hard and I hope that they'll, indeed, make the benchmarking tools available in the next release. Hopefully v2.2 will be truly production ready.

[Manubi]

Building a database without focusing on speed from the beginning is kinda strange.

[AlexFrid]

Well.. that was a long November 😅
But good news is that our benchmarking is out now: https://surrealdb.com/blog/beginning-our-benchmarking-journey

[TheOrdinaryWow]

And as for now 2.0 has released, and I am so into the features it provided. Yet still, was stopping me is that I am concerned about the performance. I cannot even find a single piece of data on the Internet.

[AlexFrid]

Totally understand, we are releasing performance benchmarks this month 🙌

[dougg0k]

Were those benchmarks released? If yes, where?

Any comparison between other databases like sqlite, postgres, etc?? Those would be useful to have a general perspective.

[nickchomey]

Nope. Were supposed to come in November. No acknowledgement of why they weren't or update of when they might come.

[AlexFrid]

Well.. that was a long November 😅
But good news is that our benchmarking is out now: https://surrealdb.com/blog/beginning-our-benchmarking-journey

[mahbubabbas]

@tobiemh a very important observation from side: I have developed one application using Surreal 1.x last year where all tables are schemaless and another one just couple of weeks back (Sdb 1.5.5) where all tables are schemafull. I can the see the speed of former is excellent where as the later one is much slower. I am using golang sdk.

Note: I am not able to use the latest SDB (2.0.4) as the Go sdk 0.3.0 is giving lots of issues which I have already reported but still waiting for fix.

[nickchomey]

When you say the performance of schemaless is excellent, what sorts of queries are you doing? And how do their performance compare to other dbs?

So far I've only found competitive performance with surreal when doing exceptionally simple "point" queries - retrieve specific record ID, as if it's a kv store. Relational is a disaster, and traversing a graph seems to be very slow as well.

It could be that I'm not doing it right, but documentation doesn't seem to go into ANY advanced details or best practices (eg how to use indexes in graph queries, or how to structure graph queries, or use query planner/explainer, etc) It's all just very basic and contrived.