Monday, November 12, 2012

Serializing Dart classes with mirrors

Serializing objects with reflection

I've updated the JsonObject library to start using the mirrors library to allow serialization from an instance of an object (and child objects, lists of objects and maps of objects) to a JSON string.
This blog post (valid at build 14669) discusses some basic use-cases of the mirror functionality that were used to create the JSON object functionality.

To show what can be achieved, the following code represents a typical use case: Converting a Person and a list of Address instances into JSON text:

import 'package:json_object/json_object.dart';

class Person {
 String name; // simple field

 var _age; // private field
 int get age => _age; // getter
 set age(value) => _age = value; // setter

 List<Address> addresses = new List<Address>();
}

class Address {
  String line1;
  String zipcode;
  Address(this.line1, this.zipcode); // constructor
}

void main() {
  var person = new Person();
  person.name = "Mr Smith";
  person.age = 30;
  person.addresses.add(new Address("1 the street", "98765");
  person.addresses.add(new Address("2 a road", "87654");

  var json = objectToJson(person);  // Here is the magic

  print(json); // Valid JSON
}

The objectToJson() top level function within the json_object library uses reflection to iterate through each of the fields and getters of an object (ignoring those that are private or static), and adds each value to an equivalent Map.  Every time it encounters a complex object (ie, not one of String, num, bool, null), or a List or Map, it recurses into itself to convert the entity into another Map.  Once it has constructed the map, it calls JSON.stringify() and returns the final string.

For the following example, we will use a simplified version of the Person class, containing only the age getter/setter pair, and the name field:

class Person {
 String name; // simple field

 var _age; // private field
 int get age => _age; // getter
 set age(value) => _age = value; // setter
}

Let's see how to serialize this to JSON.

Starting to reflect

You can incorporate the reflection library by using importing the dart:mirrors library.  We will also need the JSON library.

import "dart:mirrors"; 
import "dart:json";

This will give you a warning at present that it's not fully implemented. For this use case you can ignore it. This gives you access to the top level function reflect(), into which you pass an instance of an object, for example: reflect(person);
This, like looking in a real mirror, shows a reflection of your instance, in the form of an InstanceMirror.  You use an InstanceMirror to examine the specific instance of your object.  The instance of your object is  accessible via the InstanceMirror.reflectee property:


// Create an instance of an object
Person person = new Person();
person.name = "Mr Smith";
person.age = 30;

var instanceMirror = reflect(person); // Get an instance mirror
print(instanceMirror.reflectee == person); // true
print(instanceMirror.reflectee.name);  // "Mr Smith";
print(instanceMirror.reflectee.age);  // 30;

Accessing values dynamically

So far, we've not done anything particularly fancy. Our ultimate goal is to access each of the values dynamically, so that we can store the field names (as keys) and values in a dynamically populated map.  We'll call our map objectMap.
To access the fields by string, you can pass the field name into the getField() method of the InstanceMirror.  This returns a Future for "variables" (in Mirror parlance) such as person.name, and getters, such as person.age.

var objectMap = new Map<string,dynamic>();
map["name"] = instanceMirror.getField("name"); 
map["age"] = instanceMirror.getField("age"); 
print(objectMap); // {name: Instance of '_FutureImpl@0x127eafe4', 
                  //  age: Instance of '_FutureImpl@0x127eafe4'}

As you can see, the getField() function returns a Future rather than the actual value (as pointed out in the comments - calls to getField() are currently implemented in a synchronous fashion, however the fact that they return futures means that we should deal with them as though they are async). Our task is to convert the property values that will ultimately appear in the Future's onComplete handler into a map of key, value pairs that we can serialize to JSON. Fortunately, the Futures library contains a wait(List<Future>) function, that we can use to wait for all the futures to complete.

The following refactored code sample performs the following steps:
1. Use the onComplete event of each future to store the completed value in the objectMap
2. Use the wait() function to wait for all values to be retrieved
(But we are not quite finished)...

var objectMap = new Map();

var nameFuture = instanceMirror.getField("name");  // retrieve a Future

// when the future completetes, insert the retrieved value into the map
nameFuture.onComplete((futureValue) => objectMap["name"] = futureValue.value);

var ageFuture = instanceMirror.getField("age"); 
ageFuture.onComplete((futureValue) => objectMap["age"] = futureValue.value);

// Wait for the futures to complete
Futures.wait([nameFuture,ageFuture]).then((f) { // List of Futures

  print(objectMap); // {name: InstanceMirror on <'Mr Smith'>, 
                   //  age: InstanceMirror on <30>}
});

We are still not quite done, as the Future does not return the the class's property value - remember, we are dealing with the reflection, of a Person, not the person itself.  When the futures complete, they each return an instance mirror.  As noted above, we can access the reflected item by looking at the InstanceMirror's reflectee property, as shown in the refactored code below:

var objectMap = new Map();

var nameFuture = instanceMirror.getField("name");  // retrieve a Future
// when the future completetes, insert the retrieved value into the map
nameFuture.onComplete((futureValue) {
  var actualValue = futureValue.value.reflectee; // access the actual value 
                                                 // from the InstanceMirror
  objectMap["name"] = actualValue; // store the actual value
}); 

var ageFuture = instanceMirror.getField("age"); 
// shorthand version below:
ageFuture.onComplete((futureValue) => objectMap["age"] = futureValue.value.reflectee);

// Wait for the futures to complete
Futures.wait([nameFuture,ageFuture]).then((f) { // List of Futures

  print(objectMap); // {name: Mr Smith, age: 30}
  print(JSON.stringify(objectMap)); // Valid JSON: 
                                    // {"name":"Mr Smith","age":30}
});

Now that you have accessed the person.name and person.age values from a standard Dart class, using string values to access the properties, we need to make this a little more dynamic.  What we really need is a list of these strings, so that we can iterate through them calling getField() on each.  That's next...

Dynamically accessing the members

When working with mirrors, you can find out information about you InstanceMirror by examining it's type property.  The type property returns a ClassMirror. Unlike the InstanceMirror (which is concerned with the specific instance), the ClassMirror can provide you with information about the Class, such as its fields.  
The ClassMirror has two Map properties that we are interested in: .getters and .variables.  The keys are the field names, and the values are instances of  VariableMirror or MethodMirror.  The getters map returns MethodMirror, and the variables map returns VariableMirror, which we will soon use to find out important information about our fields.  For the moment, let's refactor our code to dynamically populate the object map based upon getters (of which the person.age is one):

var objectMap = new Map();

var futureValuesList = new List();

var classMirror = instanceMirror.type;
classMirror.getters.keys.forEach((key) {
  var future = instanceMirror.getField(key);
  future.onComplete((futureValue) {
    var actualValue = futureValue.value.reflectee; // extract the actual value
    objectMap[key] = actualValue; // store the actual value in the map, keyed with
                                  // the same name as the getter
  });
  futureValuesList.add(future); // add the future to the list.
});

// Wait for the futures to complete
Futures.wait(futureValuesList).then((f) { // List of Futures (contains age).

  print(objectMap); // {age: 30}
  print(JSON.stringify(objectMap)); // Valid JSON: {"age":30}
});

There is only one getter, so we also need to iterate through the keys of the variables map of the ClassMirror.  This presents us with a small problem, as the variables map also contains private variables, such as _age (which we have already dealt with by using the getters map.  The following line outputs the names of each of the variable keys that are reflected from the Person class:

  classMirror.variables.forEach((key,value) => print(key)); // _age
                                                            // name

Each of the values are instances of VariableMirror, and we can use VariableMirror (and likewise,  MethodMirror from the getters map if we so desire) to find out if the variable is a private member by checking the isPrivate flag.  We only want to serialize to JSON properties that are not private, so we will add an if statement to only use the public keys from the variables map.

classMirror.variables.forEach((key,VariableMirror value) {
  if (!value.isPrivate) { // only get the value if it is public
    var future = instanceMirror.getField(key);
    future.onComplete((futureValue) {
       objectMap[key] = futureValue.value.reflectee;
    });
    futureValuesList.add(future); // add the future to the list.
  }
});

And that's it. We can now serialize a simple class.

 Full Sample Code

Let's take one final look at the complete code (you should be able to paste this into the Dart Editor and run it.

import "dart:mirrors";
import "dart:json";

class Person {
 String name; // simple field

 var _age; // private field
 int get age => _age; // getter
 set age(value) => _age = value; // setter
}

void main() {
  var person = new Person();
  person.name = "Mr Smith";
  person.age = 30;
  var instanceMirror = reflect(person);
  print(instanceMirror.reflectee == person); // true
  print(instanceMirror.reflectee.name);  // "Mr Smith";
  print(instanceMirror.reflectee.age);  // 30;

  var objectMap = new Map();

  var futureValuesList = new List();

  var classMirror = instanceMirror.type;

  classMirror.getters.keys.forEach((key) {
    var future = instanceMirror.getField(key);
    future.onComplete((futureValue) {
      var actualValue = futureValue.value.reflectee; // extract the actual value
      objectMap[key] = actualValue; // store the actual value in the map, keyed with
      // the same name as the getter
    });
    futureValuesList.add(future); // add the future to the list.
  });

  classMirror.variables.forEach((key,VariableMirror value) {
    if (!value.isPrivate) { // only get the value if it is public
      var future = instanceMirror.getField(key);
      future.onComplete((futureValue) {
        objectMap[key] = futureValue.value.reflectee;
      });
      futureValuesList.add(future); // add the future to the list.
    }
  });

  // Wait for the futures to complete
  Futures.wait(futureValuesList).then((f) { // List of Futures

  print(objectMap); // {age: 30}
  print(JSON.stringify(objectMap)); // Valid JSON: 
                                    // {"name":"Mr Smith", "age":30}
  });
}

JsonObject's objectToJson() function

JsonObject uses code similar to that demonstrated above to serialize Lists of objects, Maps of objects, and objects containing other objects (or lists and maps of objects).  By using recursion, it is able to convert deeply nested hierarchies of objects to a JSON string.  Take a look at the unit tests on github for some examples, or the full source code in the mirror_based_serializer.dart class.  This has a top-level function called objectToSerializeable(), which returns something (such as a map or list) that can be serialized by JSON.stringify(), and is called by the objectToJson() function that lives in the main json_object library file.
Next steps for me is to add deserialization of a JSON string into a real class.  But that's the subject of another blog post.

But...

The only downside (at present) with using mirrors, is that it doesn't work with dart2js - so this is restricted to server-side or Dartium code only.  See this post on the Dartlang google groups for more information.

Further reading



5 comments:

  1. Are you sure Futures.wait "waits for the futures to complete"?
    According to the docs it just returns a Future that will complete when all the futures complete... so it should be Futures.wait([f1, f2]).then((_) { ..... }).

    ReplyDelete
    Replies
    1. thanks - something in the back of my mind thought that something wasn't right... I'll update the article

      Delete
  2. Nelson is right. There is no way to block in JS, so there will be no way to block in Dart. Once a function depends on an async call, it will have to be async itself.

    The code above works because invoke(), getField() and setField() are actually synchronous internally for the moment. If you're willing to rely on that you can just write getField(f).value.reflectee

    ReplyDelete
    Replies
    1. Somehow I knew that but forgot when the function appeared to work. My fault for not checking the api docs for wait()! I'll update the article.

      Delete
  3. Ok, Per Nelson and Justins comments I've updated the article to use wait().then - sorry for any confusion!

    ReplyDelete