Plugins and Configuration¶
SMQTK provides generic plugin and introspective configuration mixin classes to support interface implementation discovery and their translation form/to JSON as a plain-text configuration format.
While these two mixins function independently and can be utilized on their own, their combination is symbiotic and allows for users of SMQTK algorithms and representations to create tools in terms of interfaces and leave the specific selection of implementations for configuration time.
Later, we will introduce the two categories of configurable and (usually) pluggable class classes found within SMQTK.
Plugins¶
Motivation: We want to be able to define interfaces to generic concepts and structures around which higher order tools can be defined without strictly catering themselves to any particular implementation, while additionally allowing freedom in implementation variety without overly restricting implementation location.
In SMQTK, this is addressed via the get_plugins()
function and the Pluggable
abstract mixin class.
Interfaces and Implementations¶
Classes that inherit from the Pluggable
mixin are considered either pluggable interfaces or plugin implementations
depending on whether they fully implement abstract methods.
Interface implementations bundled within SMQTK are generally defined alongside their parent interfaces. However, other sources, e.g. other python packages, may expose their own plugin implementations via setting a system environment variable or via python extensions.
The Configurable
Mixin¶
Motivation: We want generic helpers to enable serializable configuration for classes while minimally impacting standard class development.
SMQTK provides the Configurable
mixin class
as well as other helper utility functions in smqtk.utils.configuration
for generating, and producing class instances from, configurations.
These use python’s introspect
module to determine default
configurations.
Currently this module deals in JSON for input and output configuration. Python dictionaries are used as a medium in between serialization and class input/output.
Classes that inherit from Configurable
do
need to at a minimum implement the
get_config()
instance method.
This does detract from the “minimal impact” intent of this mixin, but other
methods of allowing introspection of internal parameters require additional
structural components in the parent/implementing class.
Algorithms and Representations - The Combination¶
Interfaces found in SMQTK are generally binned into two categories: representations and algorithms.
Algorithms are interfaces to some function or operation, specifically
parameterized through their constructor and generally parameterized via the
algorithm’s interface.
The SmqtkAlgorithm
base class inherits from both
Pluggable
and
Configurable
mixins so that all descendents
gain access to the synergy they provide.
These are located under the smqtk.algorithms
sub-module.
Representations are interfaces to structures that are intended to specifically
store some sort of data structure.
Currently, the SmqtkRepresentation
only inherits
directly from Configurable
, as there are
some representational structures which desire configurability but to which
variable implementations do not make sense (like
DescriptorElementFactory
).
However most sub-classes do additionally inherit from
Pluggable
(like
DescriptorElement
).
These are located under the smqtk.representation
sub-module.
Implementing a Pluggable Interface¶
The following are examples of how to add and expose new plugin implementations for existing algorithm and representation interfaces.
SMQTK’s plugin discovery allows for exposure of plugin implementations in 3 ways:
- Parallel in location to parent interface.
- Python module path of implementation model included in the
SMQTK_PLUGIN_PATH
environment variable (see reference for formatting).- An entrypoint in a python package’s
setup.py
.
Within SMQTK¶
A new interface implementation within the SMQTK source-tree is generally implemented or exposed parallel to where the parent interface is defined.
As an example, we will show how to create a new implementation for the
Classifier
algorithm
interface.
This interface is defined within SMQTK at, from the root of the source tree,
python/smqtk/algorithms/classifier/_interface_classifier.py
.
We will create a new file, some_impl.py
, that will be placed in the
same directory with the intention that our new plugin will be picked up based
on parallel locality to the parent interface class.
We’ll define our new class, lets call it SomeImpl
, in a file
some_impl.py
:
python/
└── smqtk/
└── algorithms/
└── classifier/
├── ...
├── _interface_classifier.py
├── some_impl.py # new
In this file we will need to define the SomeImpl
class and all parent
class abstract methods in order for the class to satisfy the definition of an
“implementation”:
from smqtk.algorithms import Classifier
class SomeImpl (Classifier):
"""
Some documentation for this specific implementation.
"""
# Abstract methods from Pluggable.
# (Classifier -> SmqtkAlgorithm -> Pluggable)
@classmethod
def is_usable(cls):
...
# Our implementation-specific constructor.
def __init__(self, paramA=1, paramB=2):
...
# Abstract methods from Configurable.
# (Classifier -> SmqtkAlgorithm -> Configurable)
def get_config(self):
return {
"paramA": ...,
"paramB": ...,
}
# Classifier's abstract methods.
def get_labels(self):
...
def _classify_arrays(self, array_iter):
...
With all abstract methods defined, this implementation should now be included
in the returned set of implementation types for the parent
Classifier
interface:
>>> from smqtk.algorithms import Classifier
>>> Classifier.get_impls()
set([..., SomeImpl, ...])
SomeImpl
above should also be all set for configuration because of it
defining get_config()
and because
it’s constructor is only anticipating JSON-conpliant types.
If more complicated types are desired by the constructor the additional methods
would need to be overriden/extended as defined in the
smqtk.utils.configuration
module.
More Complicated Implementations¶
Interface-parallel implementation discovery also allows for nested sub-modules.
This is useful when an implementation requires specific or extensive support
utilities.
The __init__.py
file of the sub-module should at least expose concrete
implementation classes that should be exported as attributes for the plugin
discovery to find.
For example, such a nested sub-module implementation might look like the
following on the filesystem:
python/
└── smqtk/
└── algorithms/
└── classifier/
├── ...
├── some_impl.py # from above
└── other_impl/ # new
└── __init__.py # new
Within another python package¶
When implementing a pluggable interface in another python package, the proper
method of export is via a package’s entrypoint specifications using the
smqtk_plugins
key.
For example, let’s assume that a separate python package, OtherPackage
we’ll call it, defines a
Classifier
-implementing sub-class
OtherClassifier
in the module OtherPackage.other_classifier
.
This module location can be exposed via the package’s setup.py
entrypoints metadata, using the smqtk_plugins
key, like the following:
from setuptools import setup
...
setup(
...
entry_points={
'smqtk_plugins': 'my_plugins = OtherPackage.other_classifier'
}
)
If the other module had multiple sub-modules in which SMQTK plugins were
defined the entry_points['smqtk_plugins']
entry may instead be a list:
setup(
...
entry_points={
'smqtk_plugins': [
'classifier_plugins = OtherPackage.other_classifier',
'other_plugins = OtherPackage.other_plugins',
]
}
)
Reference¶
smqtk.utils.configuration
¶
Helper interface and functions for generalized object configuration, to and from JSON-compliant dictionaries.
While this interface and utility methods should be general enough to add JSON-compliant dictionary-based configuration to any object, this was created in mind with the SMQTK plugin module.
Standard configuration dictionaries should be JSON compliant take the following general format:
{
"type": "one-of-the-keys-below",
"ClassName1": {
"param1": "val1",
"param2": "val2"
},
"ClassName2": {
"p1": 4.5,
"p2": null
}
}
The “type” key is considered a special key that should always be present and it
specifies one of the other keys within the same dictionary. Each other key in
the dictionary should be the name of a Configurable
inheriting class type.
Usually, the classes named within a block inherit from a common interface and
the “type” value denotes a selection of a specific sub-class for use, though
this is not required property of these constructs.
-
class
smqtk.utils.configuration.
Configurable
[source]¶ Interface for objects that should be configurable via a configuration dictionary consisting of JSON types.
-
classmethod
from_config
(config_dict, merge_default=True)[source]¶ Instantiate a new instance of this class given the configuration JSON-compliant dictionary encapsulating initialization arguments.
This base method is adequate without modification when a class’s constructor argument types are JSON-compliant. If one or more are not, however, this method then needs to be overridden in order to convert from a JSON-compliant stand-in into the more complex object the constructor requires. It is recommended that when complex types are used they also inherit from the
Configurable
in order to hopefully make easier the conversion to and from JSON-compliant stand-ins.When this method does need to be overridden, this usually looks like the following pattern:
class MyClass (Configurable): @classmethod def from_config(cls, config_dict, merge_default=True): # Optionally guarantee default values are present in the # configuration dictionary. This statement pairs with the # ``merge_default=False`` parameter in the super call. # This also in effect shallow copies the given non-dictionary # entries of ``config_dict`` due to the merger with the # default config. if merge_default: config_dict = merge_dict(cls.get_default_config(), config_dict) # # Perform any overriding here. # # Create and return an instance using the super method. return super(MyClass, cls).from_config(config_dict, merge_default=False)
This method should not be called via super unless an instance of the class is desired.
Parameters: Returns: Constructed instance from the provided config.
Return type:
-
get_config
()[source]¶ Return a JSON-compliant dictionary that could be passed to this class’s
from_config
method to produce an instance with identical configuration.In the most cases, this involves naming the keys of the dictionary based on the initialization argument names as if it were to be passed to the constructor via dictionary expansion. In some cases, where it doesn’t make sense to store some object constructor parameters are expected to be supplied at as configuration values (i.e. must be supplied at runtime), this method’s returned dictionary may leave those parameters out. In such cases, the object’s
from_config
class-method would also take additional positional arguments to fill in for the parameters that this returned configuration lacks.Returns: JSON type compliant configuration dictionary. Return type: dict
-
classmethod
get_default_config
()[source]¶ Generate and return a default configuration dictionary for this class. This will be primarily used for generating what the configuration dictionary would look like for this class without instantiating it.
By default, we observe what this class’s constructor takes as arguments, turning those argument names into configuration dictionary keys. If any of those arguments have defaults, we will add those values into the configuration dictionary appropriately. The dictionary returned should only contain JSON compliant value types.
It is not be guaranteed that the configuration dictionary returned from this method is valid for construction of an instance of this class.
Returns: Default configuration dictionary for the class. Return type: dict >>> # noinspection PyUnresolvedReferences >>> class SimpleConfig(Configurable): ... def __init__(self, a=1, b='foo'): ... self.a = a ... self.b = b ... def get_config(self): ... return {'a': self.a, 'b': self.b} >>> self = SimpleConfig() >>> config = self.get_default_config() >>> assert config == {'a': 1, 'b': 'foo'}
-
classmethod
-
smqtk.utils.configuration.
cls_conf_from_config_dict
(config, type_iter)[source]¶ Helper function for getting the appropriate type and configuration sub-dictionary based on the provided “standard” SMQTK configuration dictionary format (see above module documentation).
Parameters: Raises: - This may be raised if:
- type field not present in
config
. - type field set to
None
- type field did not match any available configuration in the given config.
- Type field did not specify any implementation key.
- type field not present in
Returns: Appropriate class type from
type_iter
that matches the configured type as well as the sub-dictionary from the configuration. From this return,type.from_config(config)
should be callable.Return type:
-
smqtk.utils.configuration.
cls_conf_to_config_dict
(cls, conf)[source]¶ Helper function for creating the appropriate “standard” smqtk configuration dictionary given a Configurable-implementing class and a configuration for that class.
This very simple function simply arranges a class, using its __name__ property, and an associated dictionary into a normal pattern used for configuration in SMQTK:
>>> class SomeClass (object):
… pass >>> cls_conf_to_config_dict(SomeClass, {0: 0, ‘a’: ‘b’}) == { … ‘type’: ‘SomeClass’, … ‘SomeClass’: {0: 0, ‘a’: ‘b’} … } True
Parameters: - cls (type[Configurable]) – A class type implementing the Configurable interface.
- conf (dict) – SMQTK standard type-optioned configuration dictionary for the given class and dictionary pair.
Returns: “Standard” SMQTK JSON-compliant configuration dictionary
Return type:
-
smqtk.utils.configuration.
configuration_test_helper
(inst, config_ignored_params=frozenset(), from_config_args=())[source]¶ Helper function for testing the get_default_config/from_config/get_config methods for class types that in part implement the Configurable mixin class. This function also tests that
inst
’s parent class type’sget_default_config
returns a dictionary whose keys’ match the constructor’s inspected parameters (except “self” of course).This constructs 3 additional instances based on the given instance following the pattern:
inst-1 -> inst-2 -> inst-3 -> inst-4
This refers to
inst-2
andinst-4
being constructed from the config frominst
, andinst-3
being constructed from the config ofinst-2
. The equivalence of each instance’s config is cross-checked with the other instances. This is intended to check that a configuration yields the same class configurations and that the config does not get mutated by nested instance construction.This function uses assert calls to check for consistency.
We return all instances constructed in case the caller wants to make additional instance integrity checks.
Parameters: - inst (Configurable) – Configurable-mixin inheriting class to test.
- config_ignored_params (set[str]) – Set of parameter names in the instance type’s constructor that are
ignored by
get_default_config
andfrom_config
. This is empty by default. - from_config_args (tuple) – Optional additional positional arguments to the input
inst.from_config
method after the configuration dictionary.
Returns: Instance 2, 3, and 4 as described above.
Return type:
-
smqtk.utils.configuration.
from_config_dict
(config, type_iter, *args)[source]¶ Helper function for instantiating an instance of a class given the configuration dictionary
config
from available types provided bytype_iter
via theConfigurable
interface’sfrom_config
class-method.args
are additionally positional arguments to be passed to the type’sfrom_config
method on return.Example: >>> from smqtk.representation import DescriptorElement >>> example_config = { … ‘type’: ‘DescriptorMemoryElement’, … ‘DescriptorMemoryElement’: {}, … } >>> inst = from_config_dict(example_config, DescriptorElement.get_impls(), … ‘type-str’, ‘some-uuid’) >>> from smqtk.representation.descriptor_element.local_elements import DescriptorMemoryElement >>> isinstance(inst, DescriptorMemoryElement) True
Raises: - ValueError –
- This may be raised if:
- type field not present in
config
. - type field set to
None
- type field did not match any available configuration in the given config.
- Type field did not specify any implementation key.
- type field not present in
- AssertionError – This may be raised if the class specified as the configuration type,
is present in the given
type_iter
but is not a subclass of theConfigurable
interface. - TypeError – Insufficient/incorrect initialization parameters were
specified for the specified
type
’s constructor.
Parameters: Returns: Instance of the configured class type as specified in
config
and as available intype_iter
.Return type: - ValueError –
-
smqtk.utils.configuration.
make_default_config
(configurable_iter)[source]¶ Generated default configuration dictionary for the given iterable of Configurable-inheriting types.
For example, assuming the following simple class that descends from
Configurable
, we would expect the following behavior:>>> class ExampleConfigurableType (Configurable): ... def __init__(self, a, b): ... ''' Dummy constructor ''' >>> make_default_config([ExampleConfigurableType]) == { ... 'type': None, ... 'ExampleConfigurableType': { ... 'a': None, ... 'b': None, ... } ... } True
Note that technically
ExampleConfigurableType
is still abstract as it does not implementget_config
. The above call tomake_default_config
still functions because we only use theget_default_config
class method and do not instantiate any types given to this function. While functionally acceptable, it is generally not recommended to draw configurations from abstract classes.Parameters: configurable_iter (collections.Iterable[type]) – An iterable of class types class types that sub-class Configurable
.Returns: Base configuration dictionary with an empty type
field, and containing the types and initialization parameter specification for all implementation types available from the provided getter method.Return type: dict[str, object]
-
smqtk.utils.configuration.
to_config_dict
(c_inst)[source]¶ Helper function that transforms the configuration dictionary retrieved from
configurable_inst
into the “standard” SMQTK configuration dictionary format (see above module documentation).For example, with a simple DataFileElement: >>> from smqtk.representation.data_element.file_element import DataFileElement >>> e = DataFileElement(filepath=’/path/to/file.txt’, … readonly=True) >>> to_config_dict(e) == { … “type”: “DataFileElement”, … “DataFileElement”: { … “filepath”: “/path/to/file.txt”, … “readonly”: True, … “explicit_mimetype”: None, … } … } True
Parameters: c_inst (Configurable) – Instance of a class type that subclasses the Configurable
interface.Returns: Standard format configuration dictionary. Return type: dict
smqtk.utils.plugin
¶
Helper interface and function for implementing class discovery.
- Plugins may be accessed by one of the following ways:
Be defined within SMQTK next to their interface.
The environment variable
SMQTK_PLUGIN_PATH
may be set to a number of :-separated (; on Windows) python module paths to where plugin classes are defined.Other installed python packages may define one or more extensions for the namespace “smqtk_plugins”. This should be a single or list of extensions that specify modules within the installed package where plugins for export are implemented. Note that we desire modules, not objects, for our extensions.
For example:
... entry_points={ "smqtk_plugins": "my_package = my_package.plugins" ] ...
Or:
... entry_points = { "smqtk_plugins": [ "my_package_mode_1 = my_package.mode_1.plugins", "my_package_mode_2 = my_package.mode_2.plugins", ] } ...
-
exception
smqtk.utils.plugin.
NotUsableError
[source]¶ Exception thrown when a pluggable class is constructed but does not report as usable.
-
class
smqtk.utils.plugin.
Pluggable
[source]¶ Interface for classes that have plugin implementations
-
classmethod
get_impls
(warn=True, reload_modules=False)[source]¶ Discover and return a set of classes that implement the calling class.
See the
get_plugins
function for more details on the logic of how implementing classes (aka “plugins”) are discovered.The class-level variables
PLUGIN_ENV_VAR
andPLUGIN_HELPER_VAR
may be overridden to change what environment and helper variable are looked for, respectively.Parameters: warn (bool) – If we should warn about module import failures.
Parameters: reload_modules (bool) – Explicitly reload discovered modules from source. Returns: Set of discovered class types descending from type interface_type
andsmqtk.utils.plugin.Pluggable
whose keys are the string names of the class types.Return type: set[type[Pluggable]]
-
classmethod
is_usable
()[source]¶ Check whether this class is available for use.
Since certain plugin implementations may require additional dependencies that may not yet be available on the system, this method should check for those dependencies and return a boolean saying if the implementation is usable.
- NOTES:
- This should be a class method
- When an implementation is deemed not usable, this should emit a
- warning detailing why the implementation is not available for use.
Returns: Boolean determination of whether this implementation is usable. Return type: bool
-
classmethod
-
smqtk.utils.plugin.
get_plugins
(interface_type, env_var, helper_var, warn=True, reload_modules=False)[source]¶ Discover and return classes implementing the given
interface_class
.Discoverable implementations may either be located in sub-modules parallel to the definition of the interface class or be located in modules specified in the environment variable
env_var
.In order to specify additional out-of-scope python modules containing interface-class implementations, additions to the given environment variable must be made. Entries must be separated by the standard PATH separating character based on the operating OS standard (e.g. ‘;’ (for windows) or ‘:’ for most everything else). Entries should be importable python module paths.
When looking at module attributes, we only acknowledge those that start with an alphanumeric character. ‘_’ prefixed attributes are effectively hidden from discovery by this function when merely scanning a module’s attributes.
We required that the base class that we are checking for also descends from the
Pluggable
interface defined above. This allows us to check if a loaded classis_usable
.Within a module we first look for a helper variable by the name provided, which can either be a single class object or an iterable of class objects, to be specifically exported. If the variable is set to None, we skip that module and do not import anything. If the variable is not present, we look at attributes defined in that module for classes that descend from the given base class type. If none of the above are found, or if an exception occurs, the module is skipped.
Parameters: - interface_type (type) – Interface class type of which we want to discover implementations of (the plugins).
- env_var (str) – String name of an environment variable defining additional python module paths, whose child modules are searched for implementing sub-classes of the specified base type.
- helper_var (str) – Name of the expected module helper attribute.
- warn (bool) – If we should warn about module import failures.
- reload_modules (bool) – Explicitly reload discovered modules from source instead of taking a potentially cached version of the module.
Returns: Set of discovered class types descending from type
interface_type
andsmqtk.utils.plugin.Pluggable
whose keys are the string names of the class types.Return type:
Reload Use Warning¶
While the smqtk.utils.plugin.get_plugins()
function allows for reloading
discovered modules for potentially new content, this is not recommended under
normal conditions.
When reloading a plugin module after pickle
serializing an instance of
an implementation, deserialization causes an error because the original class
type that was pickled is no longer valid as the reloaded module overwrote the
previous plugin class type.