How to Use Huey in Your Django Project for Asynchronous Task Execution

asynchronous django huey task

If you want to boost the performance and efficiency of your Django project, one approach you could take is to use a task queue for asynchronous task execution. This means that instead of executing tasks in a blocking manner, where each task must be completed before the next one can begin, you can use a task queue library to add tasks to a queue and have them executed in the background while your Django application continues to handle incoming requests. This can improve the responsiveness and scalability of your application, allowing it to handle more traffic and complex workloads without becoming slow or unresponsive.

Huey or Celery?

Huey is a popular task queue library that is easy to use and integrates well with Django. It is written in Python and is based on the Redis in-memory data store. Celery, on the other hand, is a more powerful and feature-rich library that can be used with Django and other Python web frameworks.

You may choose Huey over Celery if you are looking for a task queue library that is user-friendly. Huey is specifically designed to be simple and intuitive, and it offers a straightforward API for scheduling and managing tasks. This makes it an excellent choice for developers who are new to task queue libraries or who are working on small-scale projects that do not require advanced task queue functionality. Overall, Huey's simplicity and ease of use make it an appealing option for developers who want a task queue library that is easy to set up and use.

Installing

To add Huey to your Django project, the first step is to install the huey and redis Python packages. Add them to your requirements.txt file or install them directly with PIP using pip install huey redis.

Next, add huey.contrib.djhuey to your INSTALLED_APPS list in your Django settings module (commonly in yourporject/settings.py). You can fine tune your Huey configuration by modifying the HUEY variable in settings. By default, it looks like this:

HUEY = {
    'huey_class': 'huey.RedisHuey',  # Huey implementation to use.
    'name': settings.DATABASES['default']['NAME'],  # Use db name for huey.
    'results': True,  # Store return values of tasks.
    'store_none': False,  # If a task returns None, do not save to results.
    'immediate': settings.DEBUG,  # If DEBUG=True, run synchronously.
    'utc': True,  # Use UTC for all times internally.
    'blocking': True,  # Perform blocking pop rather than poll Redis.
    'connection': {
        'host': 'localhost',
        'port': 6379,
        'db': 0,
        'connection_pool': None,  # Definitely you should use pooling!
        # ... tons of other options, see redis-py for details.

        # huey-specific connection parameters.
        'read_timeout': 1,  # If not polling (blocking pop), use timeout.
        'url': None,  # Allow Redis config via a DSN.
    },
    'consumer': {
        'workers': 1,
        'worker_type': 'thread',
        'initial_delay': 0.1,  # Smallest polling interval, same as -d.
        'backoff': 1.15,  # Exponential backoff using this rate, -b.
        'max_delay': 10.0,  # Max possible polling interval, -m.
        'scheduler_interval': 1,  # Check schedule every second, -s.
        'periodic': True,  # Enable crontab feature.
        'check_worker_health': True,  # Enable worker health checks.
        'health_check_interval': 1,  # Check worker health every second.
    },
}

Check out the official Huey documentation for details on what each of these settings does. We will just mention on the immediate setting. In the above snippet it is set to True if we're in a debugging environment, which means that Huey tasks will run synchronously in development. If this is set to False, Huey expects to be able to push tasks to a redis server, and for those tasks to run you need to run the consumer in a separate process by executing python manage.py run_huey.

The above steps are probably unnecessary in a development environment. We'll cover how to deploy Huey with your Django project to a production environment in a separate post.

Asynchronous Tasks

With Huey configured, you can now start using it to execute tasks asynchronously in your Django app. To do this, you'll need to define a task and register it with Huey. A task is simply a Python function that you want to execute asynchronously. Here is an example of how you might use a Huey task in a Django view:

from django.http import HttpResponse
from huey.contrib.djhuey import task


@task()
def my_async_task(request):
    # Do something asynchronously, such as send a request to an external API or send an email
    pass

def my_view(request):
    # In the view, schedule the async task to run
    my_async_task(request=request)
    return HttpResponse(...)

In this example, my_async_task is a Huey task that can be called asynchronously from within a Django view. In the my_view view, we schedule the my_async_task to run, passing in the request as an argument.

Database tasks

In Huey, a db_task is a subclass of task that is designed to be executed against a database.

The main difference between a task and a db_task is that a db_task has access to the Django database, which means that it can perform operations on the database and interact with Django models. A task, on the other hand, does not have direct access to the database, so it cannot perform operations on the database or interact with Django models directly.

Here is an example of defining a task and a db_task in Huey:

from huey import task
from huey.contrib.djhuey import db_task

from my_app.models import MyModel

@task()
def my_task(param1, param2):
    # Do something with param1 and param2
    pass

@db_task()
def my_db_task(param1, param2):
    # Do something with param1 and param2 using the Django database like
    objects=MyModel.objects.filter(name=param1).all()
    pass

Periodic tasks

Huey also can be used to schedule and execute periodic tasks. The periodic_task decorator is used to define a function that will be run as a periodic task. This function will be executed at regular intervals, according to the schedule specified when the task is created.

For example, let's say you have a Django project with a cleanup function that you want to run every day at 11 PM.You could use the periodic_task decorator to define this function as a periodic task like this:

from huey import crontab
from huey.contrib.djhuey import periodic_task

@periodic_task(crontab(hour=23, minute=0))
def cleanup():
    # Code to perform daily cleanup tasks goes here.

In this example, the cleanup function is decorated with @periodic_task, and the crontab function is used to specify the schedule for the task. In this case, the schedule is set to run the cleanup function every day at 23:00 or 11 PM.

Once the periodic task is defined, Huey will automatically execute it according to the specified schedule. This can be a useful way to perform regular maintenance or other tasks on a Django project.

If your periodic task requires access to the database, you can use db_periodic_task.

Conclusion

In summary, Huey is a powerful and easy-to-use task queue library that can be integrated into your Django project to enable asynchronous task execution. By adding Huey to your project, you can improve the performance and efficiency of your app by offloading long-running tasks to the task queue.