A tutorial details the construction, in Python, of a JSON serialization process capable of handling custom data types, tolerating malformed data, and adapting to schema evolution. The goal is to circulate data sets between files, processes, microservices and APIs without the pipeline stopping at the first anomaly.
According to the article, only a small portion of a machine‑learning engineer’s time is spent writing the model architecture. The rest is devoted to data plumbing (“data plumbing”), i.e., cleaning, preprocessing and transforming data. Part of this work involves serializing data, converting it to text to store or transmit it, and then deserializing it, i.e., rebuilding it into Python objects.
The initial problem stems from the limitations of the JSON format, which only knows a few basic types. An experiment configuration dictionary containing an Enum, a set, a Path or a datetime triggers an error as soon as json.dumps() is called, with the message “TypeError: Object of type ModelType is not JSON serializable”.
A custom encoder
The first step is to define a class that inherits from json.JSONEncoder and overrides its default() method. For each problematic type, an isinstance() check specifies the conversion to perform. Datetimes are thus transformed into a dictionary containing an ISO‑formatted value, sets into lists, Paths into strings, and Enums into their value.
One detail is central: the encoder adds a “__type__” key to objects that might lose their original nature, such as a set converted to a list. Once passed to json.dumps() via the cls argument, the sample dictionary is serialized without error, and the datetime appears as “2026-09-28T14:26:10.328310”.
An error‑tolerant decoder
In the opposite direction, the robust_decoder function is passed to json.loads() via the object_hook argument. It is called on each nested dictionary and reconstructs the original object when the “__type__” key is present. The conversions are placed in a try block that catches ValueError and TypeError exceptions.
In case of malformed data, the function logs a warning via the logging module and returns the raw value, without halting processing. The article notes that this behavior can be replaced by a fallback value or a specific error dictionary. After deserialization, the date is correctly recognized as a datetime and the list as a set.
Managing schema evolution
Model metadata schemas change frequently: hyperparameters are added, others dropped, some metrics renamed. To address this, the author proposes a factory function, load_experiment_config(), which loads the JSON with the decoder and then applies default values using get().
The example concerns a hyperparameter “learning_rate” missing from the old configuration, to which the default value 0.001 is assigned. The same principle can be applied to any new parameter. The article also mentions the possibility of removing obsolete fields during loading. Applied to the original configuration, the loading restores the expected types: an integer for the ID, a set for the variables, a PosixPath for the path, a datetime for the timestamp and a float for the learning rate.
