Sometimes, you just don’t want to see how deep the rabbit hole goes. One of those times is when you need to store a structured,nested object in a timeseries DB like InfluxDb.

The need

In my specific use case, a set of Protobf messages sent through a message bus have to be intercepted and logged in a timeseries for further analysis. To do that, the data structure must be flattened in some way before it can be stored successfully. Furthermore, especially when processing arrays, in some cases this is not acceptable to default to array index as the key of the node (for example when processing an array of objects and you want to take the object id as the key, instead of the array index). For that reason it is possible that some nodes of the structure have to be modified before being flattened.

To recap, the expectations are:

  1. To take as input a Golang map of the structured object. The map may contain nested maps or slices thas have to be flattened in a recursive way. It is up to te client to perform the conversion from other formats (json, protobuf, etc) to the map.
  2. To output a map with the flattened structure. It is up to client to permorm any further conversion or serialization.
  3. Flatten the structure until the leaf nodes (scalar values) are reached, joining the nested object keys into a unique string. It has to be possible to choose the separator used to join the string.
  4. When flattening a node, it has to be possible to perform a custom processing of the node to change / tweak both key and value using a previosly registered transformer function.
  5. When flattening a node, it has to be possible to stop the flattening of the node children / fields an leave them as a structured map.

Non goals

  1. use concurrent workers to speed up the flattening processing
  2. feature to un-flatten the structure

Flatr

FLATR Golang package implements the concepts above and it is available on Github. It uses a simple stack implementation to flatten a map and support custom transformers (some already implemented) to modify the current node before flattening it. Give it a try!