Add json/yaml serialization

[ceball]

Recently, PR #153 was opened, showing the desire for json/yaml serialization of parameterized objects.

My reading of #153's serialization:

1. does not appear to include information about the class (when you deserialize, I think you have to pass in the class)
2. does not include non-parameter attributes
3. does not provide a way to control serialization
4. assumes you only want to write to a file

I think 1, 3, and 4 are necessary to address.

[jbednar]

1. I think the YAML/JSON file definitely needs to include the class name somehow for any subobject; not sure how to best express that in that format.
2. I'm not certain whether it's useful to include non-parameter attributes, but probably not -- this mechanism is not for serializing the state of a live object, but for saving a specification by which that object could be instantiated just the same as if that object had been specified using a Python constructor call (which would also not currently allow the attributes to be set by the user, only within the constructor.
3. I'm not sure what you mean?
4. The code should definitely support streams, not just named files.

[ceball]

Re. 3: e.g. a class can control how it's pickled by implementing __getstate__() (see https://github.com/ioam/param/wiki/Serialization#controlling-serializationdeserialization).

[jbednar]

Ah, you mean that 3. it does not provide classes a way to control their own serialization, or for users to control serialization of specific classes.

Ok, yes, that's needed for pickling; not sure if it's needed for JSON/YAML. Seems like something that can be added once it's clearly required.

[ceball]

Guess I'm thinking about e.g. numbergen.RandomDistribution, which will contain a random.Random object. How would you expect that to appear in json? I think we'll have to provide some way to customize json serialization from the outset just to support what's already in param.

[ceball]

Hmm, sticking with random generators as an example to illustrate something I'm not clear about...what exactly is serialization?

Pickle presumably saves a random generator's state (where possible) so it carries on generating where it left off.

Right now script_repr omits random generators. But maybe it should (I think?) give a representation that would start the same sequence from the beginning again (i.e. capture the parameters of the random generator like seed etc), because that seems most useful given the aim of script_repr. However, that doesn't seem like serialization (it's not capturing the object's state).

What about json/yaml? For #153, where I think the serialization is for some kind of file-based config system, doing the same thing as script_repr above would make sense to me. But I can imagine wanting to serialize the current state of a parameterized object to pass via json. Sure, probably nobody outside python could use that in the case of a python random generator, but it seems closer to the definition of 'serialization'.

I guess I'm just trying to say that different users of json serialization might expect different things: some might expect something more like what pickle does, while others might expect something more like what script_repr does.

[jbednar]

It would be great if script_repr included a specification for the random number generation that would start the sequence from the beginning, and the same is true for this JSON-based config generation and reading.

Trying to serialize the full state to/from JSON/YAML would be a much bigger job, which can be postponed until there's a specific application that would require such a rich (and probably unreadable) specification whose requirements would drive the implementation. Such rich specification could be enabled specifically, using a flag asking it to represent all state, once we support it, so we can safely ignore it for now.

[ceball]

Just wanted to note YAML's support for specifying types and how you could already use it (sort of) for a configuration file.

While you can write:

a: 1.0

You can also write:

a: !!float 1

YAML contains various tags like !!int, !!set, !!map, etc. Libraries like PyYAML also add support for various python-specific tags (http://pyyaml.org/wiki/PyYAMLDocumentation#YAMLtagsandPythontypes).

For example.py containing:

import param

def fn1(a,b):
    return a*b

def fn2(a,b):
    return a+b

class Thing(param.Parameterized):
    a = param.Parameter(default=1)

class SomeThing(Thing):
    b = param.Number(default=0.5,bounds=(0,1))
    c = param.Tuple(default=(1,2))
    
class Config(param.Parameterized):
    param1 = param.Callable(default=fn1)
    param2 = param.ClassSelector(Thing,default=SomeThing())
    param3 = param.Parameter(Thing)

You can load this config.yml with PyYAML:

!!python/object:example.Config
param1: !!python/name:example.fn2
param2: !!python/object:example.SomeThing
  b: 0.75
  c: !!python/tuple [1, 2]
param3: !!python/name:example.Thing

You could also use YAML tag directives:

%TAG !o! tag:yaml.org,2002:python/object:
%TAG !n! tag:yaml.org,2002:python/name:
---
!o!example.Config
param1: !n!example.fn2
param2: !o!example.SomeThing
  b: 0.75
  c: !!python/tuple [1, 2]
param3: !n!example.Thing

>>> yaml.load(open('config.yaml'))
Config(name='Config', param1=<function fn2 at 0x1143dfd08>, param2=SomeThing(a=1, b=0.75, c=(1, 2), name='SomeThing'), param3=<class 'example.Thing'>)

Note: YAML also specifies that a single ! represents a local/application-specific tag.

[ceball]

I'd like some feedback about how e.g. a yaml config file might look.

Short version

If you have somemod.py containing:

import param
import numpy

def fn(a,b):
    return a*b

class Thing(param.Parameterized):
    thing_param = param.Parameter(default=1)

class SomeThing(Thing):
    something_number = param.Number(default=2)
    something_array = param.Array(default=numpy.array([2,2,2]))
    something_list = param.List(default=[2,2])
    something_bool = param.Boolean(default=False)
    something_string = param.Boolean(default="two")
    
class MoreThing(Thing):
    morething_list = param.List(default=[3,3,3])
    morething_param = param.Parameter(default=(3,3))
    morething_tupl = param.Tuple(default=(3,3,3,3))
    morething_call = param.Callable(default=fn)
    morething_array = param.Array(default=numpy.array([3,]))
    def __init__(self,a,**params):
        self.a = a
        super(Thing,self).__init__(**params)

class Config(param.Parameterized):
    display_name = param.String()
    conf_list = param.List()
    conf_objsel = param.ObjectSelector(default=SomeThing())
    conf_clssel = param.ClassSelector(Thing,default=SomeThing())
    conf_param1 = param.Parameter(Thing)
    conf_param2 = param.Parameter(MoreThing(a=4))
    conf_none = param.Parameter(None)
    conf_num_none = param.Number(default=None)

How does the following yaml example look for creating some instances, specifying some parameters? Crazy?

!somemod.Config
conf_param2: !somemod.MoreThing
  args*: [40, 40]
  morething_tupl: !tuple [30, 30, 30, 30]
  morething_array: !numpy.array [[30, 30],[30,30]]
conf_num_none: 40.0
conf_clssel: !somemod.MoreThing
  args*: [41, 41]
  morething_call: !somemod.fn
  morething_tupl: !tuple [31, 31, 31, 31]
  morething_array: !numpy.array [31,31,31]
conf_param1: !somemod.SomeThing
  something_number: 20
  something_list: [20, 20, 20]
  something_bool: true
  something_string: twenty

Long version

As I noted in a previous comment you can already yaml.load() the following:

!!python/object:somemod.Config
conf_param2: !!python/object/apply:somemod.MoreThing
  args: [40, 40]
  kwds:
    morething_tupl: !!python/tuple [30, 30, 30, 30]
    morething_array: !!python/object/apply:numpy.array
      args: [[[30, 30],[30,30]]]
conf_num_none: 40.0
conf_clssel: !!python/object/apply:somemod.MoreThing
  args: [41, 41]
  kwds:
    morething_call: !!python/name:somemod.fn
    morething_tupl: !!python/tuple [31, 31, 31, 31]
    morething_array: !!python/object/apply:numpy.array [[31,31,31]]
conf_param1: !!python/object:somemod.SomeThing
  something_number: 20
  something_list: [20, 20, 20]
  something_bool: true
  something_string: twenty

(You could also shorten the tags using the TAG directive.)

Above I'm using some built-in pyyaml tags (http://pyyaml.org/wiki/PyYAMLDocumentation#YAMLtagsandPythontypes). To make an instance with kwargs only:

!!python/object:module.Class { attribute: value, ... }

To make an instance with args and kwargs:

!!python/object/apply:module.function
args: [argument, ...]
kwds: {key: value, ...}

(or !!python/object/apply:module.function [argument, ...] if no kwargs).

I think we could shorten the above by registering a custom "constructor" as a fallback for unhandled tags (i.e. all single-!/"local" tags). The shortened yaml (shown earlier) could look something like this:

!somemod.Config
conf_param2: !somemod.MoreThing
  args*: [40, 40]
  morething_tupl: !tuple [30, 30, 30, 30]
  morething_array: !numpy.array [[30, 30],[30,30]]
conf_num_none: 40.0
conf_clssel: !somemod.MoreThing
  args*: [41, 41]
  morething_call: !somemod.fn
  morething_tupl: !tuple [31, 31, 31, 31]
  morething_array: !numpy.array [31,31,31]
conf_param1: !somemod.SomeThing
  something_number: 20
  something_list: [20, 20, 20]
  something_bool: true
  something_string: twenty

That is, for "local" (unhandled-by-pyyaml) tags:

• a "scalar node" represents a name (e.g. !module.Class for module.Class)
• a "sequence node" represents a name that should be instantiated with args (e.g. !tuple [] for tuple())
• a "mapping node" represents a name that should be instantiated with args and/or kwargs (e.g. !param.Parameterized {} for param.Parameterized()

The above wouldn't stop someone from using 'standard' pyyaml tags as well or instead, if they prefer.

Note that args* can't be a python kwarg, which allows it to be used in a mapping node without conflicting with any possible existing key. Alternatively, we could instead allow only kwargs to be specified (i.e. params) for single-! tags, which would make things simpler - but quite a lot of useful things couldn't then be used as easily.

[jbednar]

Even the long version looks good to me, but the short version seems very readable and suitable for humans writing a config file. As long as the custom constructor can be written in a very general way that we wouldn't have to revisit, it seems worth it to me!

[sdrobert]

Hello. Can I weigh in a little bit?

I think that serialization is a bit of overkill for these file types. Encoding things like bounds, choices, etc. becomes excessively verbose. When I use these sorts of configuration files, I just want a way to store and retrieve parameter values with some trivial conversion from a string. I probably know what Parameterized class these key-value pairs belong to. For example, I might make a class like:

class WindowParams(param.Parameterized):
    window_colour = param.ObjectSelector('blue', objects=['blue', 'red', 'yellow'])
    window_width = param.Integer(1024, bounds=(1, None))
    # ...

Then configure it with a file like

left_window:
  window_colour: red

right_window:
  window_colour: blue

And I just want to find a way to load the specified options into a couple of WindowParams instances. I think this is mostly consistent with the use case proposed in #153.

The great thing about this strategy is that it's very easy to implement over a lot of config-like file types, including YAML, JSON, INI (ConfigParser), and even command-line arguments. All that's needed are standard ways to read values from and write values to strings. Then you can have some functions like

def read_params_from_yaml(yml, parameterized, section=None):
     # ...

def write_params_to_yaml(yml, parameterized, section=None):
     # ...

Note that by passing the config object as an argument, there's no need to include any external dependencies.

If this sounds useful, I'd be happy to write it up as a PR, as I've written much of this up already for my own internal use. I'd need to know where to put stuff, though.

Thanks for your time,
Sean