1. Polymorph Data Introduction
2. Polymorph Data Encoding
3. Polymorph Data Language

• Polymorph Data Primary Design Goals
• Polymorph Data Features
• Polymorph Data Use Cases
• Polymorph StreamDB
• Polymorph Data Types
  • Atomic Data Types
  • Composite Data Types
• Polymorph Data vs. Other Data Formats

Polymorph Data Encoding (PDE) is a binary data format that is attempting to eliminate many of the limitations of other popular data formats such as XML, JSON, CSV, MessagePack, CBOR, Protobuf, Avro etc. By eliminating these limitations Polymorph Data Encoding becomes applicable in a wider variety of use cases. That way you do not have to make a data encoding choice for every single use case, but can stick with the same data encoding for far more use cases.

Polymorph Data Encoding is an advanced data format designed for use in encoding of messages in network protocols, encoding of structured data during data exchange, and as structured data encoding for file storage. Furthermore, Polymorph Data Encoding has been designed to be able to function as a binary record stream encoding for use in record stream storage as well as record stream transport via a network protocol.

Since Polymorph Data Encoding is binary it can be hard to read and edit in a text editor. To alleviate this problem I have created a textual version of PDE which is called Polymorph Data Language (PDL). You can convert a PDE file to PDL and open it in a text editor for inspection. Or, you can edit a PDL file and convert it to PDE for efficient use in your applications.

The term Polymorph Data refers to both Polymorph Data Encoding (PDE) and Polymorph Data Language (PDL) as a single data expression mechanism.

Polymorph Data Primary Design Goals

The Polymorph Data Encoding (PDE) is designed according to the following design goals:

• High versatility
• High read and write performance
• Compact encoding - small size

Polymorph Data Language is designed to be able to represent PDE consistently, and somewhat concisely, while also being reasonably easy and fast to parse. This is a trade-off, where neither will probably be achieved 100% - but both will be achieved reasonably well (80-90%) and will be improved over time.

Polymorph Data Features

These design goals led to the Polymorph data format with the following features:

• High Versatility
  • Self describing - no schema needed.
  • Many commonly used data types predefined.
  • Able to contain raw binary data.
  • Object data type (key -> value - like properties).
  • Cyclic + acyclic object graph support.
  • Tabular data type for CSV or DB result set style data.
  • Able to contain both single data fields or streams of fields.
  • Highly extensible - with custom fields.
    
    
• Compact encoding
  • TLLV encoding with compact encodings for most data types.
  • Compact tabular data and tree structures via nested tables.
  • Copy or reference fields located earlier in a data block (e.g. a file or message) to reduce data redundancy.
    
    
• Fast reading
  • Fast and easy to decode.
  • Streamable reading.
  • Partial readability.
  • Arbitrary hierarchical navigation.
    
    
• Fast Writing
  • The compact encoding means less bytes to write resulting in faster write performance.
  • The simple encoding requires fewer instructions to write than more complex encodings.

Polymorph Data Use Cases

The features of Polymorph Data might be eaiser to understand if you know what use cases they are designed for. Thus, here is a brief list of the use cases Polymorph Data is designed for:

• Structured data encoding for data structures such as objects, trees, graphs, tables and streams.
  • Object graph structures - such as in XML and JSON files.
    • Unnamed fields like in XML or inside JSON arrays.
    • Key-value pairs of fields like in JSON objects or XML attributes.
    • Acyclic + cyclic object graphs.
    • Efficient tabular structures with a single set of column header names - such as in CSV files.
    • Efficient tree structures - via tables nested in tables.
      
      
• Data encoding for streams of records that are easily appendable and replayable.
  • Change or transaction log files.
  • Event log files.
  • Time series data.
    
    
• Data encoding for data files
  • Document style files - like rich text documents, presentations and other rich media documents.
  • Record style files - like CSV files.
  • Stream style files.
  • Application log files.
  • Configuration files.
    
    
• Data encoding for network messages - containing structured data (object graphs, record sets or record streams).
  • Easy replication of record streams across a network.
  • Subscribe-notify style network communication - e.g. subscribe to a time series data set or RSS style feed.
  • Broadcast and multicast of structured media data.
  • P2P network routing
  • Smart media
  • Edge computing
  • IoT
    
    
• Subscription to data streams - via subscribe-notify style network communication.
  Each record in a stream has an offset - from which you can resume subscription.
  
  
• Incremental replication of data sets - via record streams - both locally on the same machine, and via network.
  Each record in a stream has an offset - from which you can resume replication.
  
  

Polymorph StreamDB

The Java toolkit used to work with PDE and PDL will also contain a tool called Polymorph StreamDB - which is a stream database. StreamDB can be used to query streams of PDE fields efficiently - both via a query language, and a stream processing API. StreamDB is planned to be able to query streams both stored locally, but also on remote StreamDB instances, or services with a StreamDB compatible interface.

Exactly how StreamDB lands feature-wise is still somewhat uncertain, but I have lots of ideas for features that can reasonably easily be implemented on top of PDE.

Polymorph Data Types

Polymorph Data is encoded as fields. Each field has a data type. Polymorph Data field types fall into two categories:

• Atomic types
• Composite types

An atomic type is a field that contains a single value. For instance, a single number, text, date etc.

A composite type is a field that contains other fields nested inside it.

Atomic Data Types

Polymorph Data can express the following core atomic data types:

Composite Data Types

Polymorph Data can express the following core composite data types :

While there are upper limits of the size of each individual field, there is no upper size limit for a file or stream of fields.

Polymorph Data vs. Other Data Formats

Polymorph data is designed to be competitive with the following data formats:

• MessagePack
• CBOR
• RION
• ION (from Amazon)
• Protobuf
• Avro

MessagePack, CBOR, RION, ION and Polymorph Data all use a similar TLLV encoding. Polymorph uses an encoding that is more similar to MessagePack and CBOR, while having all the advanced data types of RION.

I have not yet benchmarked Polymorph Data, but performance should be similar to that of MessagePack, CBOR, RION and ION. The encoding is in som parts a bit simpler than RION, so perhaps read and write speed is a little bit higher, but I will have to verify that before making any certain claims about that.