Home » Publications » Caching to Improve GeoWeb Reliability

The Problem

The OGC distributed services model of having each organization run its own web services has a number of advantages and is being embraced world wide. A major problem, however, is its reliability.

While some organizations have great IT departments committed to keeping their services highly available, some of the best data resides with organizations that struggle to keep their services live. This problem is exacerbated for centralized aggregators, like Statens kartverk or ESDIN's forthcoming aggregation services, as their service will appear less reliable if the cascaded services are not themselves reliable. This problem is most apparent with Web Map Service (WMS) delivery, where Google and other commercial players have set high availability and performance standards.

The Solutions

The solution favored by those with central authority is to require a certain level of uptime and reliability. This works reasonably well when there is political will and money to back it up, but it can be easily complemented by more bottom-up solutions to achieve higher levels of service.

The halfway solution is “provider caching” — making copies of the relevant information and serving them elsewhere. The OGC is in the process of adopting a Web Map Tiling Service (WMTS) specification that will extend the WMS specification to create tiles that are more easily cached.

OpenGeo has been working on GeoWebCache, a caching solution that works with any WMS service, and there are several other similar projects. Caching services sit between a WMS and the client, not only greatly accelerating the speed but increasing the reliability as well.

If a back-end WMS service goes down then the cache still works. And since the caching engine is a much simpler and less resource intensive product it has greater overall availability. It can also be clustered easily for even higher availability. Requiring partners to install a cache in addition to their default service will ensure a greater degree of reliability.

The friendliest bottom-up solution with the lowest barrier to entry for data-serving partners is for an aggregator to provide them with centralized caches. This incentivizes partners to go through the aggregator, since it not only increases the reliability of their service but also reduces the load on their servers. The centralized cache is best if it is quite large, but it could also leverage techniques to only cache the most popular tiles requested in order to take the most stressful load off of partners.

The Product

A few key extensions to GeoWebCache could turn it into an ideal product for allowing an aggregator to cache partner WMS services, evolving from a provider cache to an aggregator cache. This service could take the cascading WMS services and start caching them to the maximum allowed disk space while keeping the most used tiles available for everyone.

To work with cascading WMS providers the key component is to allow their administrators to truncate the cache by notifying the aggregator cache when they change the data or styles of their services and triggering a re-cache. Partners should have control over the cache in several ways: they should be able to log in and manually expire the cache by drawing an area of the map and selecting their layer; optionally, they should be able to limit the rate at which the cache is seeded so as not overload their service.

In addition, the cache should provide automated notifications, such as GeoRSS feeds, to allow dynamic services to broadcast what has changed and what the cache needs to update. This will work well with implementations like GeoServer, which has a Geosynchronization module in the OGC OWS-5 testbed that sends notifications of transactions.

Extending GeoWebCache to provide sophisticated reporting of statistics that document usage, report load patterns, and identify areas of interest would allow the cascading WMS providers to make informed decisions about infrastructure requirements and other priorities. In addition to the aggregate statistics, the results would be easily visualized as a heat map overlay on top of the original data.

GeoWebCache is a great base to build upon, as it already has a solid caching core that can use various back-ends, different cache expiry mechanisms, a REST API to programmatically control what is cached, and a basic user interface. The needed extensions would be:

• Friendly user interface to allow partners to log in and truncate, seed, update their caches
• Improved security integration, handling of multiple users with per layer permissions
• WMTS specification implementation
• Truncation of cache based on GeoRSS feed notifications
• Clustering / lateral cache synchronization / increased reliability
• Queued requests to back-end servers to control load
• Statistics collection hooks and presentation modules

Together these would create a great back-end interface that provides a real win to partners who would get clear value out of registering their services with a centralized catalog or aggregator. A OpenGeo Suite Enterprise contract would easily suffice in providing the amount of work needed to accomplish these improvements.

Other White Papers