Update Django template fragment caches in the background


The title of this post is what I originally tried to do, but the solution to the problem was to do a small refactoring and to use the low level cache API. This post will explain how I modified my approach after thinking about the problem for a while.


I was working on a site that relies heavily on template fragment caching. Over time the code behind those template fragments had become more and more complex, and rendering the template fragments took more and more time. One problem with template fragment caching in Django is that it's not easily possible to rebuild them in the background, by default they are rendered during a request.

def mytag(instance, context_var):
    return _my_expensive_function(instance, context_var)

The template tag above simply dumps the data into the template which performs some non-trivial rendering logic. This is not ideal as there is no good place to cache anything except in the template itself.

Below is the first step towards my solution.

def mytag(instance, context_var):
    data = _my_expensive_function(instance, context_var)
    return render_to_string(                                                                     
         "templatetags/data_rendering.html", {"complex_data": data}                     

The refactoring as shown above is really simple: I moved the template fragment into its own file and rendered it from inside the template tag. The template fragment was put into the file templatetags/data_rendering.html. Once this refactoring worked I could simply cache the rendered template fragment.

from django import template
from django.core.cache import cache
from django.template.loader import render_to_string

register = template.Library()

def mytag(instance, context_var, use_cache=True):
    cache_key = "my_fragment_{}_{}".format(instance.pk, context_var)
    if use_cache is True:
        rendered = cache.get(cache_key)
        if rendered:
            return rendered
    data = _my_expensive_function(instance, context_var)
    rendered = render_to_string(                                                                     
         "templatetags/data_rendering.html", {"complex_data": data}                     
    cache.set(cache_key, rendered, 3700)
    return rendered

This is still a minimal change and pretty much standard code when working with the Django caching framework. The problem of rebuilding the fragment in the background is not yet solved, one last piece is missing, and there are many solutions to this problem. I went with a custom management command that's called by cron, but anything that can run a task periodically will do.

from django.core.management.base import BaseCommand
from myapp.templatetags.mytags import mytag
from myapp import models, context_var_choices

class Command(BaseCommand):
    def handle(self, *args, **options):
        for instance in models.MyModel.objects.to_cache():
            for context_var in context_var_choices:
                mytag(instance, context_var, use_cache=False)

Now all I had to do was to call the command every hour as the cache is valid for more than one hour. Since then I've adopted this pattern on multiple sites and they load much faster than before because the caches are always pre-filled when a request happens. Basic cache invalidation happens by keeping an mtime field on every model and in the cache key.


Start a new thread

Cancel reply
Markdown. Syntax highlighting with <code lang="php"><?php echo "Hello, world!"; ?></code> etc.
DjangoPythonBitcoinTuxDebianHTML5 badgeSaltStackUpset confused bugMoneyHackerUpset confused bugX.OrggitFirefoxWindowMakerBashIs it worth the time?i3 window managerWagtailContainerIrssiNginxSilenceTorWorldInternet securityFontGnuPGThunderbirdJenkins