Designing Maps/Dictionaries

Roto
1

We now have lists, so the obvious next step is maps/dictionaries. Before we rush into implementing that, however, we should first figure out the design.

Syntax

Recall that Roto currently already has this syntax for anonymous records:

let x = {
    foo: 10,
    bar: 20,
};

That would (in my opinion) also be the most intuitive syntax for maps, so we’ll have to figure something out there.

Maybe we don’t need special syntax, after all, Rust doesn’t have that either. It does feel like it would fit a scripting language to have some syntax for it though.

I’ve been thinking to give anonymous records the following syntax:

let x = @{
    foo: 10,
    bar: 20,
};

Or maybe that’s what maps should be! The @ symbol is used in some other languages for lists as well.

And yet another question is whether there should be syntax for indexing maps. I don’t think we need that in a first version though.

Behaviour

We also have to consider what the behaviour of the maps should be.

  • Should it (like Python) preserve insertion order?
  • More questions? Feel free to reply below!
2

I would have a question about maps! Namely, will it be possible to return a map defined on the Roto side, to Rust, without the Rust half knowing in advance anything about the map (apart from that it exists)?

I have two use cases for this, one is logging, where I construct a JSON-shaped log message on the Roto side, and the Rust side “only” has to serialize it into JSON, and emit the log.

The other use case is transferring state between Roto functions via Rust. The flow here is that Rust would call a Roto function, which returns a map. Rust would then store this map, not do anything with it, but call another Roto function (same runtime, different function) at another point of execution, maybe, with the previously returned map as the function’s input. The Rust side in this case would not need to know anything about the map, the decision whether to call the other function would not be based on it, but on external factors. I just wish to ferry some state between function calls.

Would maps implement serde’s Serialize & Deserialize? Because that would make everything I want to do with maps so very easy.

3

Excellent question! I’ll try to break it down.

In principle, this will be a very type-safe API, much like how lists currently work. So with just maps, I don’t think you’ll be able to use arbitrary shapes of data. But that doesn’t mean we can’t think about how to support that!

I have two use cases for this, one is logging, where I construct a JSON-shaped log message on the Roto side, and the Rust side “only” has to serialize it into JSON, and emit the log.

For the first use-case, I can imagine that we make a type similar to serde_json::Value that most types can be converted into. That should be flexible enough.

The other use case is transferring state between Roto functions via Rust. The flow here is that Rust would call a Roto function, which returns a map. Rust would then store this map, not do anything with it, but call another Roto function (same runtime, different function) at another point of execution, maybe, with the previously returned map as the function’s input.

It sounds like you want something like I described a long time ago in this issue: Schemas and dynamic values · Issue #62 · NLnetLabs/roto · GitHub. The big challenge is ensuring that the type that you take out of Roto is the same as the one that you put in. It might need to be a similar API to Rust’s Any, where you can try to downcast into the type you want.

Would maps implement serde’s Serialize & Deserialize? Because that would make everything I want to do with maps so very easy.

Probably not in the first version, but it’s something I’ve been think about for sure! Not just for maps but also for records and other Roto types.

1 Like