Ditto 1.1
Last month we released Ditto, a powerful new server for building communities on Nostr. This new release adds major stability and performance improvements we discovered after it became adopted by the Nostr community.
Database Timeouts
After the initial launch of Ditto, we started to see random periods where our sites would freeze (showing a black screen) for a minute or two at a time, with users unable to do anything.
This problem has been fixed in Ditto 1.1 by adding database timeouts to each query.
Timeouts are configurable by the admin with the following environment variables (in milliseconds):
DB_TIMEOUT_RELAY(default:1000)DB_TIMEOUT_DEFAULT(default:5000)DB_TIMEOUT_TIMELINES(default:15000)
It turns out this problem was being caused by Nostr clients making slow queries to Ditto's relay. Therefore, a more aggressive timeout is now applied to the relay by default, while internal queries are allowed to take longer.
This has shown us there is still work to be done regarding database performance. We found a mixture of queries that were borderline malicious, alongside ones that should not be so slow. We will work to decrease these timeouts more in the next update.
Metrics
When dealing with performance issues, one thing you should NOT do is randomly try changing things and see if it "seems" better.
During my tenure at Truth Social, I'm happy to have learned a data-driven approach. For everyone's sanity, you must fully understand the problem before attempting to "fix" it.
So Ditto now exposes a /metrics endpoint in Prometheus exposition format (also known as "OpenMetrics" format) which exposes a lot of data about Ditto over time, including: Nostr relay connections, HTTP responses, database query time, and most importantly "available database connections".
"Available database connections" is what ended up cracking the code, as we could see it drop to 0 whenever the site would freeze. Meanwhile, "database query time" would spike at the same time. So we deduced that slow queries were taking up all the connections, allowing us to introduce timeouts and finally fix the problem. After that, the graph looked perfect. So that's the value of data, folks.
You can follow the Ditto documentation to learn how to install Prometheus alongside your Ditto server, as well as useful queries to make your own graphs!
Some people have even built cool Grafana dashboards already. 👀
Rate Limiting
Rate limiting is something we should not have slept on in the first release. Ditto 1.1 now rate-limits both the HTTP API and the Nostr relay.
Ditto uses the X-Real-Ip header for IP rate limiting, so ensure your server is configured correctly otherwise rate-limiting will be disabled.
The HTTP API is limited to 300 requests per 5-minute window, while the relay allows 300 messages per 1-minute window.
Due to a bug in Deno itself, WebSocket rate-limiting did not work correctly at first. So admins should upgrade to at least Deno 1.45.0 for the best performance.
nos2x Support
Several users reported that nos2x was broken on Ditto. Up until that point I had only tested with Alby, but after pulling down nos2x locally I discovered that it wasn't injecting the script into the page as quickly as it could. So I submitted a patch to nos2x (merged) and another patch to nos2x-fox to fix it.
I also patched Soapbox to be more resilient to this issue, so upgrading either the browser extension or Soapbox (if you're a Ditto admin) should fix it.
Top Zaps
When viewing zaps on posts, the highest zaps are now shown at the top, and the results are paginated.
bolt11 invoices are now also decoded, so zap receipts without any amount tag will still get displayed correctly.
What's Next
Ditto's goal is to serve large communities online. Before we can do that, there is still some groundwork to be done.
While 1.1 fixes basic server performance, 1.2 will fix Nostr support by switching to a custom relay pool implementation with full outbox support. This will improve our ability to pull events from other relays, as well as fix problems with follower lists getting overriden.
Special Thanks
Thank you to our team: patrickReiis, shantaram, and MK! And thank you to the Nostr community for your outpouring of support for Ditto. This is just the beginning. 🚀