Welcome to cachings’s documentation!

caching (Caching Module)

Author:

Description:

This Module supports functions and classes for caching e.g. properties of other instances.

Submodules:

Unittest:

See also the unittest documentation.

class caching.property_cache_json(source_instance, cache_filename, load_all_on_init=False, callback_on_data_storage=None, max_age=None, store_on_get=True)

See also parent property_cache_pickle for detailed information.

Important

  • This class uses json. You should only use keys of type string!

  • Unicode types are transfered to strings

See limitations of json.

Example:

#!/usr/bin/env python
# -*- coding: UTF-8 -*-

import caching
import logging
import sys
import time


class test_slow_data(object):
    DATA_VERSION = 0.1
    KEY_ONE = '1'
    KEY_TWO = '2'
    KEY_THREE = 'three'
    KEY_FOUR = 'four'
    KEY_FIVE = 'five'
    KEYS = [KEY_ONE, KEY_TWO, KEY_THREE, KEY_FOUR, KEY_FIVE]

    def data_version(self):
        return self.DATA_VERSION

    def get(self, key, default=None):
        try:
            return getattr(self, f'__{key}__')()
        except AttributeError:
            return default

    def keys(self):
        return self.KEYS

    def uid(self):
        return None

    def print_n_sleep(self, k):
        sys.stdout.write('slow get executed for %s\n' % k)
        time.sleep(3)

    def __1__(self):
        self.print_n_sleep("__1__")
        return 'one'

    def __2__(self):
        self.print_n_sleep("__2__")
        return 'two'

    def __three__(self):
        self.print_n_sleep("__three__")
        return 'three'

    def __four__(self):
        self.print_n_sleep("__four__")
        return 'four'

    def __five__(self):
        self.print_n_sleep("__five__")
        return 'five'


if __name__ == "__main__":
    # Logging configuration
    logging.basicConfig(
        format="%(asctime)s: %(levelname)8s - %(name)s - %(message)s",
        level=logging.DEBUG,
        stream=sys.stdout
    )
    # Example
    tm = time.time()
    data = caching.property_cache_json(test_slow_data(), 'cache.json')
    data.add_source_get_keys(data.KEY_THREE)
    print('Testing property_cache (json):\n--------------------------------')
    for key in data.keys():
        print(data.get(key))
    print('--------------------------------\nThe execution time was %.1fs' % (time.time()-tm))

Will result on the first execution to the following output (with a long execution time):

Testing property_cache (json):
--------------------------------
2024-09-22 00:02:43,744:    DEBUG - caching - Cache file does not exists (yet).
2024-09-22 00:02:43,745:    DEBUG - caching - cache-file stored (cache.json)
2024-09-22 00:02:43,745:    DEBUG - caching - Loading property for key='1' from source instance
slow get executed for __1__
2024-09-22 00:02:46,746:    DEBUG - caching - Adding key=1, value=one with timestamp=1726956166 to chache
2024-09-22 00:02:46,747:    DEBUG - caching - cache-file stored (cache.json)
one
2024-09-22 00:02:46,748:    DEBUG - caching - Loading property for key='2' from source instance
slow get executed for __2__
2024-09-22 00:02:49,749:    DEBUG - caching - Adding key=2, value=two with timestamp=1726956169 to chache
2024-09-22 00:02:49,750:    DEBUG - caching - cache-file stored (cache.json)
two
2024-09-22 00:02:49,751:    DEBUG - caching - Key 'three' is excluded by .add_source_get_keys(). Uncached data will be returned.
slow get executed for __three__
three
2024-09-22 00:02:52,752:    DEBUG - caching - Loading property for key='four' from source instance
slow get executed for __four__
2024-09-22 00:02:55,754:    DEBUG - caching - Adding key=four, value=four with timestamp=1726956175 to chache
2024-09-22 00:02:55,755:    DEBUG - caching - cache-file stored (cache.json)
four
2024-09-22 00:02:55,755:    DEBUG - caching - Loading property for key='five' from source instance
slow get executed for __five__
2024-09-22 00:02:58,757:    DEBUG - caching - Adding key=five, value=five with timestamp=1726956178 to chache
2024-09-22 00:02:58,758:    DEBUG - caching - cache-file stored (cache.json)
five
--------------------------------
The execution time was 15.0s

With every following execution the time cosumption my by much smaller:

Testing property_cache (json):
--------------------------------
2024-09-22 00:02:58,825:    DEBUG - caching - Loading properties from cache (cache.json)
2024-09-22 00:02:58,825:    DEBUG - caching - Providing property for '1' from cache
one
2024-09-22 00:02:58,825:    DEBUG - caching - Providing property for '2' from cache
two
2024-09-22 00:02:58,825:    DEBUG - caching - Key 'three' is excluded by .add_source_get_keys(). Uncached data will be returned.
slow get executed for __three__
three
2024-09-22 00:03:01,827:    DEBUG - caching - Providing property for 'four' from cache
four
2024-09-22 00:03:01,827:    DEBUG - caching - Providing property for 'five' from cache
five
--------------------------------
The execution time was 3.0s
class caching.property_cache_pickle(source_instance, cache_filename, load_all_on_init=False, callback_on_data_storage=None, max_age=None, store_on_get=True)

This class caches the data from a given source_instance. It takes the data from the cache instead of generating the data from the source_instance, if the conditions for the cache usage are given.

Required properties for the source_instance

  • uid(): returns the unique id of the source’s source or None, if you don’t want to use the unique id.

  • keys(): returns a list of all available keys.

  • data_version(): returns a version number of the current data (it should be increased, if the get method of the source instance returns improved values or the data structure had been changed).

  • get(key, default): returns the property for a key. If key does not exists, default will be returned.

Hint

You are able to use all parameters and methods of the source_instance identically with the property_cache instance.

Parameters:

  • source_instance (instance) – The source instance holding the data

  • cache_filename (str) – File name, where the properties are stored as cache

  • load_all_on_init (bool) – True will load all data from the source instance, when the cache will be initialised the first time.

  • callback_on_data_storage (method) – The callback will be executed every time when the cache file is stored. It will be executed with the instance of this class as first argument.

  • max_age (int or None) – The maximum age of the cached data in seconds or None for no maximum age.

  • store_on_get (bool) – False will prevent cache storage with execution of the .get(key, default) method. You need to store the cache somewhere else.

The cache will be used, if all following conditions are given

  • The key is in the list returned by .keys() method of the source_instance

  • The key is not in the list of keys added by the .add_source_get_keys() method.

  • The cache age is less then the given max_age parameter or the given max_age is None.

  • The uid of the source instance (e.g. a checksum or unique id of the source) is identically to to uid stored in the cache.

  • The data version of the source_instance is <= the data version stored in the cache.

  • The value is available in the previous stored information

Example:

#!/usr/bin/env python
# -*- coding: UTF-8 -*-

import caching
import logging
import sys
import time


class test_slow_data(object):
    DATA_VERSION = 0.1
    KEY_ONE = '1'
    KEY_TWO = '2'
    KEY_THREE = 'three'
    KEY_FOUR = 'four'
    KEY_FIVE = 'five'
    KEYS = [KEY_ONE, KEY_TWO, KEY_THREE, KEY_FOUR, KEY_FIVE]

    def data_version(self):
        return self.DATA_VERSION

    def get(self, key, default=None):
        try:
            return getattr(self, f'__{key}__')()
        except AttributeError:
            return default

    def keys(self):
        return self.KEYS

    def uid(self):
        return None

    def print_n_sleep(self, k):
        sys.stdout.write('slow get executed for %s\n' % k)
        time.sleep(3)

    def __1__(self):
        self.print_n_sleep("__1__")
        return 'one'

    def __2__(self):
        self.print_n_sleep("__2__")
        return 'two'

    def __three__(self):
        self.print_n_sleep("__three__")
        return 'three'

    def __four__(self):
        self.print_n_sleep("__four__")
        return 'four'

    def __five__(self):
        self.print_n_sleep("__five__")
        return 'five'


if __name__ == "__main__":
    # Logging configuration
    logging.basicConfig(
        format="%(asctime)s: %(levelname)8s - %(name)s - %(message)s",
        level=logging.DEBUG,
        stream=sys.stdout
    )
    # Example
    tm = time.time()
    data = caching.property_cache_json(test_slow_data(), 'cache.json')
    data.add_source_get_keys(data.KEY_THREE)
    print('Testing property_cache (pickle):\n--------------------------------')
    for key in data.keys():
        print(data.get(key))
    print('--------------------------------\nThe execution time was %.1fs' % (time.time()-tm))

Will result on the first execution to the following output (with a long execution time):

Testing property_cache (pickle):
--------------------------------
2024-09-22 00:03:01,889:    DEBUG - caching - Cache file does not exists (yet).
2024-09-22 00:03:01,890:    DEBUG - caching - cache-file stored (cache.json)
2024-09-22 00:03:01,890:    DEBUG - caching - Loading property for key='1' from source instance
slow get executed for __1__
2024-09-22 00:03:04,891:    DEBUG - caching - Adding key=1, value=one with timestamp=1726956184 to chache
2024-09-22 00:03:04,891:    DEBUG - caching - cache-file stored (cache.json)
one
2024-09-22 00:03:04,892:    DEBUG - caching - Loading property for key='2' from source instance
slow get executed for __2__
2024-09-22 00:03:07,892:    DEBUG - caching - Adding key=2, value=two with timestamp=1726956187 to chache
2024-09-22 00:03:07,893:    DEBUG - caching - cache-file stored (cache.json)
two
2024-09-22 00:03:07,894:    DEBUG - caching - Key 'three' is excluded by .add_source_get_keys(). Uncached data will be returned.
slow get executed for __three__
three
2024-09-22 00:03:10,894:    DEBUG - caching - Loading property for key='four' from source instance
slow get executed for __four__
2024-09-22 00:03:13,895:    DEBUG - caching - Adding key=four, value=four with timestamp=1726956193 to chache
2024-09-22 00:03:13,896:    DEBUG - caching - cache-file stored (cache.json)
four
2024-09-22 00:03:13,897:    DEBUG - caching - Loading property for key='five' from source instance
slow get executed for __five__
2024-09-22 00:03:16,898:    DEBUG - caching - Adding key=five, value=five with timestamp=1726956196 to chache
2024-09-22 00:03:16,899:    DEBUG - caching - cache-file stored (cache.json)
five
--------------------------------
The execution time was 15.0s

With every following execution the time cosumption my by much smaller:

Testing property_cache (pickle):
--------------------------------
2024-09-22 00:03:16,978:    DEBUG - caching - Loading properties from cache (cache.json)
2024-09-22 00:03:16,978:    DEBUG - caching - Providing property for '1' from cache
one
2024-09-22 00:03:16,978:    DEBUG - caching - Providing property for '2' from cache
two
2024-09-22 00:03:16,979:    DEBUG - caching - Key 'three' is excluded by .add_source_get_keys(). Uncached data will be returned.
slow get executed for __three__
three
2024-09-22 00:03:19,980:    DEBUG - caching - Providing property for 'four' from cache
four
2024-09-22 00:03:19,981:    DEBUG - caching - Providing property for 'five' from cache
five
--------------------------------
The execution time was 3.0s
add_source_get_keys(keys)

This will add one or more keys to a list of keys which will always be provided by the source_instance instead of the cache.

Parameters:

keys (list, tuple, str) – The key or keys to be added

full_update(sleep_between_keys=0)

With the execution of this method, the complete source data which needs to be cached, will be read from the source instance and the resulting cache will be stored to the given file.

Parameters:

sleep_between_keys (float, int) – Time to sleep between each source data generation

Hint

Use this method, if you initiallised the class with store_on_get=False

get(key, default=None)

Method to get the cached property. If the key does not exists in the cache or source_instance, default will be returned.

Parameters:

  • key – key for value to get.

  • default – value to be returned, if key does not exists.

Returns:

value for a given key or default value.

Indices and tables