WOODY'S
FINDINGS

What's new in Scout 4.0.0?

Checkout the project page

Global refactoring

Scout 4.0.0 is a global refactor of the code base. This was a necessary step to offer new features. Also, the code is now more robust, faster and more flexible to welcome new features.

New data structure

To reach this goal, a new data structure has been chosen to represent a PathExplorer values. The ExplorerValue in an indirect enum and thus is a purely functional structure. For Scout features, it allows to write less and cleaner code, but also to remove the need to manage the states a PathExplorer had in the previous versions.

This new data structure also allows to use Codable to encode and decode data, which offers several new possibilities, like customising a Coder to better fit one’s use case, or to set and add Codable types with no effort (more on that).

The XML parsing has not changed and still uses AEXML. I tried in several ways to use ExplorerValue with a XML coder but this always led to informations loss or strange behaviours. Thus I rather rewrote the XML features with this new ”functional mindset” and I believe it is clearer. Also, small new features like attributes reading are now offered.

New path parsing

The Path parsing is now done with a Parser rather than with regular expressions. This is more robust and faster. The same goes for parsing a path and its value when adding or setting a value with the command-line tool.


New features list


Breaking changes

Swift and Command-line

Command-line


ExplorerValue (Swift)

Serialisable PathExplorers like Plist, JSON and YAML or others like XML can use this type to set, add, or read a value.

The new ExplorerValue is the following enum.

public indirect enum ExplorerValue {
    case int(Int)
    case double(Double)
    case string(String)
    case bool(Bool)
    case data(Data)
    case date(Date)
    case array([ExplorerValue])
    case dictionary([String: ExplorerValue])
}

Expressible ExplorerValue

ExplorerValue implements the "Expressible" protocols when it's possible for the types it deals with. This means that it's possible to define an ExplorerValue like the following examples.

let string: ExplorerValue = "string"
let dict: ExplorerValue = ["foo": "bar"]
let array: ExplorerValue = ["Riri", "Fifi", "Loulou"]

Codable

ExplorerValue conforms to Codable. The new SerializablePathExplorer (used for JSON, Plist and XML) uses this conformance to offer initialisation from Data. But this also means that any Coder can be used to read an ExplorerValue from Data. This was already possible to use a different serialiser than the default one in the previous implementations. But customising a Coder is much simpler and now more common in Swift. For instance, setting a custom Date decoding strategy is always offered in most coders.

ExplorerValueConvertible

Setting and adding a value to an explorer now works with ExplorerValue. For instance, to set Tom’s age to 60:

json.set("Tom", "age", to: .int(60))

Of course, convenience types and functions are offered, so the line above can be written like this:

json.set("Tom", "age", to: 60)

This is made possible with the ExplorerValueRepresentable protocol. It only requires a function to convert the type to an ExplorerValue.

protocol ExplorerValueRepresentable {
    func explorerValue() throws -> ExplorerValue
}

Default implementation for primivite types

Default implementations are provided for the values mapped by ExplorerValue like String, Double, an Array if its Element conforms to ExplorerValueRepresentable and a Dictionary if its Value conforms to ExplorerValueRepresentable.
Some examples:

let stringValue = "toto"
try json.set("name", to: stringValue)
let dict = ["firstName": "Riri", "lastName": "Duck"]
try json.set("profile", to: dict)

Default implementation for Codable types

Also, a default implementation for any Encodable type is provided. An Encoder is implemented to encode a type to an ExplorerValue. Similarly, a Decoder is implemented to decode an ExplorerValue to a Decodable type with the protocol ExplorerValueCreatable. A type alias is provided to group both protocols:

ExplorerValueConvertible = ExplorerValueRepresentable & ExplorerValueCreatable

For instance with a simple struct.

struct Record: Codable, ExplorerValueConvertible {
    var name: String
    var score: Int
}

let record = Record(name: "Riri", score: 20)

It’s possible to set the record value at a specific path.

plist.set("ducks", "records", 0, to: record)

XML element

The new ExplorerXML can also set and add ExplorerValues, as well as be converted to one. Because XML is not serializable, this process might loose informations. Options to keep attributes and single child strategies are offered. This is useful in the conversion features like XML → JSON. Whenever it’s possible, ExplorerXML will keep as much information as possible. When it’s not possible, the type will act consistently. For instance, when setting a new Array value, the children of the XML element will all be named Element.

In Swift, it’s possible to rather set an AEXMLElement to have more freedom on the children. This requires more work, but I believe it’s a good thing to have this possibility. To know how to create and edit AEXMLElements, please find the repo.


Data and Date values

As part of the new ExplorerValue type, Data and Date values are supported.


Set/Add group values (Command-line)

It’s now possible to set or add a group value to a path. An array is a list of values separated by commas and enclosed by square brackets. A dictionary is a list of (key, value) parts separated by double points. Those values are separated by commas and enclosed by squared brackets.

Some examples

Set Tom colors to an array made of strings (automatically inferred).

scout set -i People.json -f json "Tom.colors=[Blue, White, Yellow]"

Set Robert weekdays to a dictionary of integers (automatically inferred).

scout set -i People.json -f json "Robert.weekdays=[monday: 5, thirsday: 3]"

It’s possible to nest values in each other. Although it’s not recommenced to nest too much, this can be useful when the value is built programmatically.

Set Robert weekdays to a dictionary of arrays

scout set -i People.json -f json \
"Robert.weekdays=[monday: [1, 2], thirsday: [3, 4]]"

Do note that type inferring still work. This is why it’s not needed in the examples to clearly specify the type. It’s still possible also to force a type, like a string rather than an integer. For instance

scout set -i People.json -f json "Robert.scores=['123', '456']"

To force a string is done by enclosing with ”/” or ”’” like /23/ or '89'. Forcing a real (Plist only) is done by enclosing with tildes ”~” like ~23~.


Empty group values

As adding a value to a path will no more create dictionaries or arrays on the fly, the new method is to add empty arrays or dictionaries. To do so, specify [] for an empty array and {} for an empty dictionary.

For instance, to add a new empty array to Robert scores.

scout set -i People.json -f json "Robert.scores=[]"

And to add a new dictionary for Robert weekdays.

scout set -i People.json -f json "Robert.weekdays={}"

Add new values

I believe this will provide a clearer and more robust interface to add new value. If it’s possible to specify arrays and dictionaries directly with the new syntax, it might be safer or easier to do it programmatically (in a loop for example).

colors=(Red White Blue)
scout add -m $file -f $format "Tom.colors=[]"

for color in $colors; do
    scout add -m $file -f $format "Tom.colors[#]=$color"
done

Work with Zsh arrays

Set arrays

It’s possible set or add an array from Zsh directly by enclosing it with square brackets.

colors=(Red White Blue)
scout add -i People.json -f json "Tom.colors=[$colorsArray]"

Set associative arrays

The same is true for associatives arrays which are a counterpart of dictionaries. To specify a dictionary to Scout, it’s possible to provide a Zsh associative array as a list of key values enclosed with curl brackets.

declare -A ducks=(Riri 10 Fifi 20 Loulou 30)
dict=${(@kv)ducks}
scout add -i $file -f $format "Tom.ducks={$dict}"

Get an array or dictionary in Zsh

Zsh integration goes in the other way with two new export functions. To get a 1-dimension array from the data, the option —export|-e can be used with the array option and by enclosing the result with brackets.

hobbies=("${(@f)$(scout read -i People.json -f json "Robert.hobbies" -e array)}")
echo $hobbies[1] # video games

The same goes for dictionaries to associative arrays with the dictionary option.

declare -A movie=("${(@f)$(scout read -i People.json -f json "Suzanne.movies[0]" -e dictionary)}")
echo $movie[title] # Tomorrow is so far

XML attributes

When working with XML and the read features, Scout will allow to read an attribute with a key element.

For instance with the following XML.

<Tom score="20">
    <height>175</height>
    <age>68</age>
</Tom>

It’s now possible to read the score value.

Command-line

scout read -i File.json -f json "Tom.score"

Swift

json.read("Tom", "score")

If an attribute is named the same way as an element, the attribute will always be read first. To read the element, it’s still possible to use an index. For instance with

<Tom score="20">
    <height>175</height>
    <age>68</age>
</Tom>

The following will output 20

Command-line

scout read -i File.json -f json "Tom.score"

Swift

json.read("Tom", "score")

When the following will output 68.

Command-line

scout read -i File.json -f json "Tom.[1]"

Swift

json.read("Tom", 1)

XML export strategies (Swift)

When exporting an ExplorerXML to an ExplorerValue, informations can be lost. It’s possible to use thekeepingAttribute parameter to keep an element attributes when exporting. When an element as attributes, the structure of the exported value will change for a dictionary holding a attributes key and a value. Also, as it’s not easy to know if one child should be exported as an array or a dictionary, Scout will look at the element’s name to decide. It’s possible to use another strategy to enforce the export as an array, a dictionary, or with a custom function.


CSV import

A new CSV import feature is available to convert a CSV input as JSON, Plist, YAML or XML. A cool feature when working with named headers is that they will be treated as paths. This can shape very precisely the structure of the converted data. For instance, the following CSV

name.first;name.last;hobbies[0];hobbies[1]
Robert;Roni;acting;driving
Suzanne;Calvin;singing;play
Tom;Cattle;surfing;watching movies

will be converted to the following Json structure.

[
  {
    "hobbies" : [
      "acting",
      "driving"
    ],
    "name" : {
      "first" : "Robert",
      "last" : "Roni"
    }
  },
  {
    "hobbies" : [
      "singing",
      "play"
    ],
    "name" : {
      "first" : "Suzanne",
      "last" : "Calvin"
    }
  },
  {
    "name" : {
      "first" : "Tom",
      "last" : "Cattle"
    },
    "hobbies" : [
      "surfing",
      "watching movies"
    ]
  }
]

When there are no headers, the input will be treated as a one or two dimension(s) array.

Swift

let json = PathExplorers.Json.fromCSV(csvString, separator: ";", hasHeaders: true)

The hasHeaders boolean is needed to specify whether the CSV string begins with named headers.

Command-line

scout csv -i people.csv -s ";" -f json --headers

The headers|--no-headers flag is needed to specify whether the CSV string begins with named headers. It’s also possible to use the standard input to provide the CSV data.