Last week we released Raphtory 0.4.0, making some major changes to the way we ingest and analyse data. With this we increased throughput to a consistent 4.6 million updates a minute (running only on a laptop) and reduced the time taken for messaging during analysis by over 10x.
Prior versions have had a long running issue with back pressure where components could get overloaded and caught in a death spiral of garbage collection. One of the key changes to solve this was us taking back control of the message queues. Moving away from the default Akka implementation (the framework we build on top of).
Sometimes, change is easy (or well easier) when you know what the problem(s) are. But it is finding the problem that’s hard. Profilers are a great tool that can help you understand precisely what is going on or at least act as a strong starting point.