In any optimization project, your path starts easy. Wandering through the fields and towns, you make progress quickly, with fellow companions trying to optimize their own projects and publishing their insights and efforts online. The experience is easy and brisk, with a logical sequence of simple fixes involving little code or effort.
As you make progress, you move past the fields and towns ascend into the forests, where the bottlenecks aren’t clearly identifiable, and the hills, where fixes are a bigger lift. The path becomes thinner, with fewer people taking the same path, and any documentation becomes more obscure and harder to understand. You start to encounter beasts: Biased and misleading benchmarks, latency-filled and incomplete monitoring tools, mind-bending architectural choices rearing their head in critical dependencies, and your own nasty house bugs that infect the increasing complex codebase you are building.
And finally, you make it to the peaks, peering your way through the blizzard, all alone, trying not to slip off into the chasm of a serious performance bug or regression. The beasts re-appear in front of you (if not dealt with conclusively). They have been evolved into dragons by the higher expectations and harsher standards, and can sweep the project into a chasm of despair. And from time to time, you can catch a a brief (and possibly illusory) glimpse at the stars of perfection through the clouds.
This is the story of such a journey. A simple start, a clear goal, and the adventures that ensued.
The project starts with a simple, clear concept: We need a fast whole slide image viewer. Whole slide images are huge, gigapixel, multi-scale images that can be zoomed into to see fine detail, zoomed out to see large structures, and pan around to see the whole image which is much larger than the screen. Its very much like interacting with google maps.
However, this simple start got rocky very fast when it came to specification and architecture: How fast is fast-enough? Are any existing slide viewers fast enough? Is it possible to build such a fast viewer with the architecture we wanted? People started throwing around numbers: Can humans notice the difference between 200ms latencies and 100ms latencies? The existing on-prem viewer is fast enough, and it gets tiles from the server in 20ms, is it necessary to replicate that network performance?
Engineering management decided to intentionally add bends and false horizons to the path so that individual contributors would not have to face this anxiety. At first there was a clear goal of: get down round trip movement->network-render round trip times to 200ms. Then when we started getting close to this, a 100ms target was revealed. Then once we started getting close to that, we started hearing about some of the anxiety around 30-40ms draw times.
We set our to meet our 200ms with confidence in our team, our infrastructure, and our plan. The path was clear. We had a good hardware vendor (AWS S3/EC2) a minimal, well managed internal stack, and a good team managing it, and leadership approval to take the time we needed to do it right.
Our chief architect laid out some high level constraints clearly to guide us. We had to put the data in the same continent. We had to ensure that the server was just a proxy and couldn’t be easily CPU throttled. We had to ensure this proxy was scalable. And then he gave us plenty of space to let the team benchmark and iterate on fixing the dumb performance bugs that inevitably come up and cause systems to be slower than they should be. And it just worked. We were getting tiles so much faster and more consistently, and the viewer was looking a lot better.
This step wasn’t easy by its nature, it was easy due to strong engineering leadership committed to prioritizing system simplicity, and an expert, well prepared infrastructure team
As soon as the network started giving us consistently fast responses, our next problem became clear—the frontend framework—and our weaknesses in leadership expertise in that area became revealed fairly quickly, and left us vulnerable.
Our existing viewer had been depending heavily on OpenSeadragon, a popular open source library for zoomable images like our own. On the surface, OpenSeaDragon was simple, dependency-free, modular and extensible javascript library. The whole thing, with bells and whistles included was only a few thousand lines of code. It had support for webgl and other modern technologies. Many of our competitors used it for their viewers. It was the obvious choice.
For a long time, it was good enough, that we never shopped around for alternatives, or poked into what it actually did, we just built around it, hooking more and more functionality into its callbacks and extending more components. Unlike the backend, where there was significant architectural push to keep things simple, the frontend complexity was allowed to grow more freely to allow for product iteration. But it meant that it turned into quite the bramble. With 50,000 lines of application code hooked closely into this framework’s specific callbacks and extensible modules, changing viewer frameworks was a painful turn to make — every change would break something.
So we placed our hopes in “tuning” or “fixing OpenSeadraon instead. But after spending time tweaking its configuration, it became increasingly clear OpenSeadragon, for all its virtues, was never meant to be a super fast viewer. Instead, it was clearly optimized for slow servers and networks: rate limiting requests, heavy animations enabled by default to make the network. These decisions weren’t just an implementation detail, they were reflected in choices in the external API. It turns out that highly “modular” design really means “architecture baked into the interface”.
So optimizing it for speed was sort of like trying to use a boat to carry your gear on mountain roads. It might look like a cart from a distance, but it just doesn’t have wheels, it was built for something else, and slapping on wheels haphazardly won’t take you far. Especially when it was already heavily with surrounding code.
Well, I’m getting ahead of myself – we didn’t realize this so clearly at the time. At the time, not knowing better or having any expertise, we gave it our best shot regardless of this missmatch.
We saw progress, but it was painful. Weeks spent banging our heads against OpenSeadragon’s 10-year old javascript with multiple authors that didn’t understand each other’s plans. Weeks measuring performance improvements on some machines that just vanished on other machines. And it was awful, and we just weren’t making steady progress, and the ultimate goal of a fast, reliable viewer was looking further and further out of reach, and the team’s motivation was getting lower and lower.
Meanwhile, progress was slow on the backend too. With the main server and data storage issues resolves, the bottleneck became various network issues. We played around with edge network acceleration technologies like AWS cloudfront and AWS global accelerator. Both worked in some cases, and were slow in others. Direct connections offered low initial latencies struggled to provide maximum bandwidth to the client. Global accelerator was generally good on many networks, but some networks could not tolerate the jumbo packets. AWS Cloudfront had decent max bandwidth and was compatible with all networks, but had expensive and unmanageable cold-hot behavior as it scaled up and down client’s connection to the edge.
Sometimes, before continuing forward, you just have to stop and rebuild. Will we spend the next few months fighting with openseadraon, or will it just be much more efficient to re-build our frontend stack from scratch. And eventually, after progress slowed to a virtual stop, leadership gave us approval to do just it.
Remember that this was a hard decision:
So, we’re looking at strictly more work and harder work. Which is why its best visualized as base-building and equipment manufacture. Its not about making progress, its about preparation.
Once the decision had landed though, progress moved fast, because we had already become experts during our fights with openseadraon:
The core architecture can be summarized as:
And just as importantly, what we decided not to do:
Between the 4 of us, we had something working in the application in about 6 weeks. And we were all working on other projects concurrently, and had other maintenance duties.
And even though it was a base camp, base camps are pretty high up — already, it was visibly the fastest cloud-based slide viewer in the world. With no hairy optimizations, just good, solid software design.
Importantly though, none of the performance improvements we saw would have been impossible with our previous approach of iterating on openseadragon. The code was simple, the ideas clear, and there was no reason openseadragon couldn’t do the same thing (with uglier and harder code). The reason we built this base camp was not to stop and sit and enjoy our simple clean system, it was to continue our adventure to the peaks of performance.
From this strong base, we could really start playing around with crazy optimizations — making the system more complex again, for the purpose of pure speed.
All of these insights were only gained at the cost of beating our head against the wall over and over and over again, trying to figure out why our application wasn’t as fast as hoped, and how it could be better.
But these would not have been possible with openseadragon — these innovations were only possible to discover due to the freedom granted by working with your own system that you built from scratch, with your own team that you trust.
One problem is that, if you push for peak performance, you start seeing worst case performance also drop.
Unfortunately, we were never really able to resolve these in the project timeframe.
And this is where it currently stands — on some commputers, this is the fastest slide viewer in the world, even compared to on-prem hosted systems — and on some systems its a laggy mess.