STEF

What is STEF?

STEF is a data format and network protocol designed for efficient sequential reading and writing of records of structured data. It is optimized for minimal payload size and high serialization speed. Think Protocol Buffers, but smaller and faster.

Key Features

• Schema-driven binary format.
• Support for primitive and composite data types.
• Columnar encoding, small payloads.
• Fast serialization and deserialization.
• Extensible schema support.
• Network protocol with forward and backward interoperability of schema.
• Open specification and reference implementations in Go and Java

Example: Define a STEF Schema for JSON-like Data

package jsonlike

// The struct with "root" attribute defines the records in a STEF stream.
struct Record root {
  // List fields in this struct, the syntax is: FieldName FieldType.
  // Each record in this STEF stream is a JsonValue.
  Value JsonValue
}

// A oneof can store one of the the listed fields.
// Empty oneof is the equivalent of "null" in JSON.
oneof JsonValue {
  Object JsonObject
  Array  []JsonValue
  String string
  Number float64
  Bool   bool
}

// A multimap is a key-value list.
multimap JsonObject {
  key   string
  value JsonValue
}

Generate Serializers

# stefgen --lang=go jsonlike.stef
Generating modifiedfields.go
Generating jsonvalue.go
Generating record.go
Generating jsonobject.go
Generating jsonvaluearray.go
Generating readerstate.go
Generating writerstate.go
Generating recordwriter.go
Generating recordwriter_test.go
Generating recordreader.go

Write STEF Records

// Prepare a memory buffer to write the STEF stream to.
buf := &pkg.MemChunkWriter{}

// Create a Writer for the JSON-like schema
w := jsonlike.NewWriter(buf, pkg.WriterOptions{})

// Build a record in memory.
writer.Record.Value.SetString("Hello, World!")

// Write the record to the stream.
writer.Write()

// Flush the stream to the buffer.
writer.Flush()

Learn More

• STEF Schema Definition Language
• STEF Specification (detailed format and protocol)
• Stefgen Code Generator (generate code from STEF schemas)
• Benchmarks (performance results)
• GitHub Repository

License