{
  "$type": "site.standard.document",
  "canonicalUrl": "https://rednafi.com/python/redis-cache/",
  "description": "Cache API responses with Redis in Python to reduce redundant requests, improve response times, and handle expiring key-value pairs efficiently.",
  "path": "/python/redis-cache/",
  "publishedAt": "2020-05-25T00:00:00.000Z",
  "site": "at://did:plc:fgtm2c26vfcj74rfmeggbyqj/site.standard.publication/3mnl6f7ob462z",
  "tags": [
    "Python",
    "API",
    "Redis"
  ],
  "textContent": "_Updated on 2023-09-11_: _Fix broken URLs._\n\nRecently, I was working with Mapbox's [Route optimization API]. It tries to solve the\n[traveling salesman problem] where you provide the API with coordinates of multiple places\nand it returns a duration-optimized route between those locations. This is a perfect usecase\nwhere [Redis] caching can come handy. Redis is a fast and lightweight in-memory database\nwith additional persistence options; making it a perfect candidate for the task at hand.\nHere, caching can save you from making redundant API requests and also, it can dramatically\nimprove the response time as well.\n\nI found that in my country, the optimized routes returned by the API do not change\ndramatically for at least for a couple of hours. So the workflow will look something like\nthis:\n\n- Caching the API response in Redis using the key-value data structure. Here the requested\n  coordinate-string will be the key and the response will be the corresponding value.\n- Setting a timeout on the records.\n- Serving new requests from cache if the records exist.\n- Only send a new request to MapBox API if the response is not cached and then add that\n  response to cache.\n\nSetting up Redis & RedisInsight\n\nTo proceed with the above workflow, you'll need to install and setup Redis database on your\nsystem. For monitoring the database, I'll be using [RedisInsight]. The easiest way to setup\nRedis and RedisInsight is through [Docker]. Here's a docker-compose that you can use to\nsetup everything with a single command.\n\nThe above docker-compose file has two services, redis and redisinsight. I've set up\nthe database with a dummy password ubuntu and made it persistent using a folder named\nredis-data in the current working directory. The database listens in localhost's port\n6379. You can monitor the database using redisinsight in port 8000. To spin up Redis and\nRedisInsight containers, run:\n\nThis command will start the database and monitor accordingly. You can go to this\nlocalhost:8000 link using your browser and connect redisinsight to your database. After\nconnecting your database, you should see a dashboard like this in your redisinsight panel:\n\n![RedisInsight dashboard showing connected Redis database overview][image_1]\n\nPreparing Python environment\n\nFor local development, you can set up your python environment and install the dependencies\nusing pip. Here, I'm on a Linux machine and using virtual environment for isolation. The\nfollowing commands will work if you're on a \\*nix based system and have python 3.12\ninstalled on your system. This will install the necessary dependencies in a virtual\nenvironment:\n\nWorkflow\n\nConnecting Python client to Redis\n\nAssuming the database server is running and you've installed the dependencies, the following\nsnippet connects redis-py client to the database.\n\nThe above excerpt tries to connect to the Redis database server using the port 6379.\nNotice, how I'm providing the password ubuntu via the password argument. Here,\nclient.ping() helps you determine if a connection has been established successfully. It\nreturns True if a successful connection can be established or raises specific errors in\ncase of failures. The above function handles AuthenticationError and prints out an error\nmessage if the error occurs. If everything goes well, running the redis_connect() function\nwill return an instance of the redis.client.Redis class. This instance will be used later\nto set and retrieve data to and from the redis database.\n\nGetting route data from MapBox API\n\nThe following function strikes the MapBox Route Optimization API and collects route data.\n\nThe above code uses Python's [HTTPx] library to make the get request. HTTPx is almost a\ndrop-in replacement for the ubiquitous [requests] library but way faster and has async\nsupport. Here, I've used context manager httpx.Client() for better resource management\nwhile making the get request. You can read more about [context managers] and how to use\nthem for hassle free resource management.\n\nThe base_url is the base url of the route optimization API and the you'll need to provide\nyour own access token in the access_token field. Notice, how the url variable builds up\nthe final request url. The coordinates are provided using the\nlat0,lon0;lat1,lon1;lat2,lon2... format. Rest of the function sends the http requests and\nconverts the response into a native dictionary object using the response.json() method.\n\nSetting & retrieving data to & from Redis database\n\nThe following two functions retrieves data from and sets data to redis database\nrespectively.\n\nHere, both the keys and the values are strings. In the second function,\nset_routes_to_cache, the client.setex() method sets a timeout of 1 hour on the key.\nAfter that the key and its associated value get deleted automatically.\n\nThe central orchestration\n\nThe route_optima function is the primary agent that orchestrates and executes the caching\nand returning of responses against requests. It roughly follows the execution flow shown\nbelow.\n\n![Flowchart showing cache check and API request decision flow][image_2]\n\nWhen a new request arrives, the function first checks if the return-value exists in the\nRedis cache. If the value exists, it shows the cached value, otherwise, it sends a new\nrequest to the MapBox API, cache that value and then shows the result.\n\nExposing as an API\n\nThis part of the code wraps the original Route Optimization API and exposes that as a new\nendpoint. I've used [FastAPI] to build the wrapper API. Doing this also hides the underlying\ndetails of authentication and the actual endpoint of the MapBox API.\n\nPutting it all together\n\nYou can copy the complete code to a file named app.py and run the app using the command\nbelow (assuming redis, redisinsight is running and you've installed the dependencies\nbeforehand):\n\nThis will run a local server where you can send new request with coordinates.\n\nGo to your browser and hit the endpoint with a set of new coordinates. For example:\n\nThis should return a response with the coordinates of the optimized route.\n\nIf you've hit the above URL for the first time, the cache attribute of the json response\nshould show false. This means that the response is being served from the original MapBox\nAPI. However, hitting the same URL with the same coordinates again will show the cached\nresponse and this time the cache attribute should show true.\n\nInspection\n\nOnce you've got everything up and running you can inspect the cache via redis insight. To do\nso, go to the link below while your app server is running:\n\nSelect the Browser panel from the left menu and click on a key of your cached data. It\nshould show something like this:\n\n![RedisInsight browser panel displaying cached coordinate data as key-value pairs][image_3]\n\nAlso you can play around with the API in the swagger UI. To do so, go to the following link:\n\nThis will take you to the swagger dashboard. Here you can make requests using the\ninteractive UI. Go ahead and inspect how the caching works for new coordinates.\n\n![FastAPI Swagger UI showing route optimization API endpoint][image_4]\n\nRemarks\n\nYou can find the complete source code of the app in my [HTTP request caching with Redis]\nrepository.\n\nDisclaimer\n\nThis app has been made for demonstration purpose only. So it might not reflect the best\npractices of production ready applications.\n\n\n\n\n[route optimization api]:\n    https://docs.mapbox.com/api/navigation/#optimization\n\n[traveling salesman problem]:\n    https://en.wikipedia.org/wiki/Travelling_salesman_problem\n\n[redis]:\n    https://redis.io/\n\n[redisinsight]:\n    https://redislabs.com/redisinsight/\n\n[docker]:\n    https://www.docker.com/\n\n[httpx]:\n    https://github.com/encode/httpx\n\n[requests]:\n    https://github.com/psf/requests\n\n[context managers]:\n    /python/contextmanager\n\n[fastapi]:\n    https://fastapi.tiangolo.com/\n\n[http request caching with redis]:\n    https://github.com/rednafi/redis-request-caching\n\n[image_1]:\n    https://blob.rednafi.com/static/images/redis_cache/img_1.png\n\n[image_2]:\n    https://blob.rednafi.com/static/images/redis_cache/img_2.png\n\n[image_3]:\n    https://blob.rednafi.com/static/images/redis_cache/img_3.png\n\n[image_4]:\n    https://blob.rednafi.com/static/images/redis_cache/img_4.png",
  "title": "Effortless API response caching with Python & Redis"
}