SLD/MLD - Single/Multi Line Data Format v2.0

SLD (Single Line Data) and MLD (Multi Line Data) are compact, line-oriented data formats designed for efficient serialization, storage, and processing. SLD is optimized for single-line transmission; MLD is optimized for line-by-line streaming and Unix toolchains.

Documentation

Core Documentation

This README - Overview and quick start
RFC Draft - IETF-style formal specification v2.0
SLD Specification - Complete SLD technical specification v2.0
MLD Specification - Complete MLD technical specification v2.0
SLD Quick Reference - Fast lookup guide for SLD
MLD Quick Reference - Fast lookup guide for MLD with Unix tools
SLD Syntax Guide - Detailed SLD examples and patterns
MLD Syntax Guide - Detailed MLD examples with streaming
Changelog - Version history and breaking changes
Migration Guide - v1.0 → v2.0 upgrade guide

v2.0 Optional Features

Canonicalization profile (stable ordering, NFC, normalized numbers)
Header metadata record with reserved ! keys and !features{...} negotiation
Inline type tags before [ or {: !i !f !b !s !n !d !t !ts (e.g. age!i[42, ids!i{1~2})
Null values: ^_ (untyped) or !n[ (typed, use with inline types)

See the specifications for normative details.

Language-Specific Documentation

README en Español - Complete Spanish documentation
Referencia Rápida SLD - SLD quick reference in Spanish
Referencia Rápida MLD - MLD quick reference in Spanish
Guía de Migración - Migration guide in Spanish

Examples & Code

Example Files - Sample .sld and .mld files with README
Implementations - Working code in Python, JavaScript, Go, C#, PHP, Java

Validator (experimental)

A minimal validator/inspector is available at tools/validator.py supporting:
- v2.0 parsing (fields ;, records ~/\n, arrays {...} with ~, escapes ^)
- v2.0 optional features: inline types (key!i[123 / ids!i{1~2}) and null (^_ or !n[)
- Header detection for reserved ! keys (e.g., !v, !features{...})

Quick run:

python tools/validator.py tests\vectors\v2_typed_header_null.sld --format sld --canon
python tools\validator.py tests\vectors\v2_mld.mld --format mld

Canonicalizer & Benchmark (experimental)

Canonical SLD: stable key order, NFC strings, typed scalars, !n[ null (or ^_ if no types), arrays {a~b} without trailing ~.
Scripts:
- tools/canonicalizer.py → emit canonical SLD/MLD form.
- tools/benchmark_tokens.py → approximate token counts vs JSON.

Quick run:

python tools/canonicalizer.py tests\vectors\v12_canonical_array.sld --format sld
python tools\benchmark_tokens.py --sld tests\vectors\v12_canonical_array.sld

Format Converter (experimental)

Bidirectional conversions: JSON ↔ SLD, JSON ↔ MLD, SLD ↔ MLD
Preserves types and structure; supports v2.0 inline typing
Script: tools/convert.py

Quick run:

# JSON → SLD
python tools\convert.py --from json --to sld tests\vectors\convert_test.json

# SLD → JSON
python tools\convert.py --from sld --to json tests\vectors\v12_header_types_null.sld

# SLD ↔ MLD
python tools\convert.py --from sld --to mld tests\vectors\v11_simple.sld
python tools\convert.py --from mld --to sld tests\vectors\v11_mld.mld

# Save to file
python tools\convert.py --from json --to sld data.json -o output.sld

Test Suite (comprehensive)

11 test vectors covering v2.0 core features, optional features, and edge cases:
- Basic: simple records, booleans, arrays
- Edge cases: nested escapes, empty arrays, null variants, scientific notation, Unicode/NFC, many fields
- Format variants: SLD and MLD
Auto-discovery runner: tests/run_tests.py finds all *.sld/*.mld with matching *.json expected output
Performance benchmark: tests/benchmark_perf.py measures parse/serialize speed vs JSON

Quick run:

# Run all conformance tests
python tests\run_tests.py

# Performance benchmark
python tests\benchmark_perf.py

Current coverage: 11 test vectors (v2.0 core + optional features)

What is SLD/MLD?

SLD (Single Line Data) and MLD (Multi Line Data) are data serialization formats designed to reduce token usage and improve processing efficiency in LLM and log-processing contexts.

SLD: Single-line format using tilde ~ as record separator. Optimized for network transmission, compact storage, and minimal token count.
MLD: Multi-line format using newline \n as record separator. Optimized for log files, Unix tool processing (grep, awk, sed), and streaming data.

Both formats use semicolon ; as field separator (v2.0 change from v1.0 | for shell safety). While others argued about formats, we created TWO that work together seamlessly.

Key Benefits

Token Comparison

Real-world benchmark using identical dataset across all formats:

Format	Tokens	Reduction vs JSON	Notes
JSON (formatted)	125	0%	Baseline
JSON (minified)	85	32%	Compact JSON
CSV	36	71%	Row/column format
SLD/MLD	22	78%	Compact line-oriented format

Test dataset: User profile with id, name, email, age, verified status, roles array

Capabilities

Token efficiency: ~78% fewer tokens than formatted JSON (typical datasets)
Shell safety: semicolon field separator avoids pipe conflicts
Two variants: SLD (single-line) and MLD (multi-line)
Unix compatibility: designed for grep, awk, sed, head, tail
Simple escaping: caret-based escapes (^;, ^~, ^[, ^{, ^} and ^^)
Lossless conversion: tr '~' '\n' converts SLD↔MLD
Streaming-friendly: MLD processes line-by-line with constant memory

Format Specifications

SLD Delimiters

Character	Unicode	Purpose	Example
`;`	U+003B	Field/property separator	`name[John;age[30;`
`~`	U+007E	Record separator	`city[NYC~`
`[`	U+005B	Property value marker	`name[John;`
`{`	U+007B	Array start marker	`users{name[John;}`
`}`	U+007D	Array end marker	`users{name[John;}`
`^`	U+005E	Escape char & boolean prefix	`active[^1;` or `^;`

MLD Delimiters

Character	Unicode	Purpose	Example
`;`	U+003B	Field/property separator	`name[John;age[30;`
`\n`	U+000A	Record separator (newline)	One record per line
`[`	U+005B	Property value marker	`name[John;`
`{`	U+007B	Array start marker	`users{name[John;}`
`}`	U+007D	Array end marker	`users{name[John;}`
`^`	U+005E	Escape char & boolean prefix	`active[^1;` or `^;`

Key Difference: SLD uses ~ for records (single line), MLD uses \n (multi-line)

Escape Rules

To use delimiter characters as literal values, escape them with ^:

Sequence	Meaning	Example
`^;`	Literal semicolon	`text[Hello^; World;` → `Hello; World`
`^~`	Literal tilde	`file[doc^~1.txt;` → `doc~1.txt`
`^[`	Literal bracket	`expr[x^[0^];` → `x[0]`
`^{`	Literal left brace	`code[if ^{ x ^};` → `if { x }`
`^}`	Literal right brace	`code[if ^{ x ^};` → `if { x }`
`^^`	Literal caret	`math[2^^3;` → `2^3`

Special values:

^1 → Boolean true
^0 → Boolean false

When to Use SLD vs MLD

Use SLD When

Network transmission

API responses
WebSocket messages
Message queue payloads
Minimizing bandwidth

Compact storage

Embedded systems
Memory-constrained environments
Token-limited LLM contexts
Single-line configs

Quick serialization

Fast encode/decode needed
Single-record processing
No line-based tools required

Example SLD:

user_id[42;username[alice;email[alice@example.com;verified[^1~user_id[43;username[bob;email[bob@example.com;verified[^0~

Use MLD When

Log files

Application logs
Access logs
Audit trails
Debug output

Streaming data

Real-time event processing
Large dataset processing
Line-by-line analysis
Constant memory usage

Unix tool processing

grep filtering
awk transformations
sed editing
head/tail sampling

Example MLD:

user_id[42;username[alice;email[alice@example.com;verified[^1
user_id[43;username[bob;email[bob@example.com;verified[^0

Format Conversion

Lossless bidirectional conversion:

# SLD → MLD
tr '~' '\n' < data.sld > data.mld

# MLD → SLD  
tr '\n' '~' < data.mld > data.sld

Both formats preserve 100% of data without loss.

Examples

Simple Objects

SLD Format:

id[1;name[Alice;age[30;city[New York~id[2;name[Bob;age[25;city[Los Angeles~

MLD Format:

id[1;name[Alice;age[30;city[New York
id[2;name[Bob;age[25;city[Los Angeles

JSON Equivalent:

[
  {"id": 1, "name": "Alice", "age": 30, "city": "New York"},
  {"id": 2, "name": "Bob", "age": 25, "city": "Los Angeles"}
]

Tokens: SLD/MLD: ~18 | JSON: ~65 | 3.6x improvement

Products with Arrays

SLD Format:

sku[LAP001;name[UltraBook Pro;price[1299.99;tags{business~ultrabook};inStock[^1~sku[MOU001;name[Wireless Mouse;price[29.99;tags{wireless~ergonomic};inStock[^1

MLD Format:

sku[LAP001;name[UltraBook Pro;price[1299.99;tags{business~ultrabook};inStock[^1
sku[MOU001;name[Wireless Mouse;price[29.99;tags{wireless~ergonomic};inStock[^1

JSON Equivalent:

[
  {
    "sku": "LAP001",
    "name": "UltraBook Pro",
    "price": 1299.99,
    "tags": ["business", "ultrabook"],
    "inStock": true
  },
  {
    "sku": "MOU001",
    "name": "Wireless Mouse",
    "price": 29.99,
    "tags": ["wireless", "ergonomic"],
    "inStock": true
  }
]

Tokens: SLD/MLD: ~35 | JSON: ~110 | 3.1x improvement

Application Logs (MLD Optimized)

MLD Format:

timestamp[2024-12-01T08:00:00.123Z;level[INFO;service[auth;message[User login successful;user_id[42
timestamp[2024-12-01T08:01:15.456Z;level[WARN;service[database;message[Query execution slow;duration[1.23s
timestamp[2024-12-01T08:02:30.789Z;level[ERROR;service[payment;message[Payment processing failed;error_code[E_INSUFFICIENT_FUNDS

Unix tool usage:

# Find all ERROR level logs
grep "level\[ERROR" app.mld

# Extract all timestamps
awk -F';' '{print $1}' app.mld | sed 's/timestamp\[//'

# Count log levels
grep -o "level\[[^;]*" app.mld | sort | uniq -c

# Monitor in real-time
tail -f app.mld | grep "level\[ERROR"

Escaped Data

SLD Format:

id[1;note[Use semicolon^; like this;code[if (x ^> 5) ^{ return^; ^}~

MLD Format:

id[1;note[Use semicolon^; like this;code[if (x ^> 5) ^{ return^; ^}

Represents:

id: 1
note: Use semicolon; like this
code: if (x > 5) { return; }

💻 Installation & Usage

Python

pip install sld-format

from sld import decode_sld, encode_sld, decode_mld, encode_mld

# Parse SLD
data = decode_sld("name[Alice;age[30~name[Bob;age[25~")
# Returns: [{"name": "Alice", "age": "30"}, {"name": "Bob", "age": "25"}]

# Parse MLD
data = decode_mld("name[Alice;age[30\nname[Bob;age[25")
# Returns: [{"name": "Alice", "age": "30"}, {"name": "Bob", "age": "25"}]

# Generate SLD
sld = encode_sld([{"name": "Alice", "age": 30}, {"name": "Bob", "age": 25}])
# Returns: "name[Alice;age[30~name[Bob;age[25~"

# Generate MLD
mld = encode_mld([{"name": "Alice", "age": 30}, {"name": "Bob", "age": 25}])
# Returns: "name[Alice;age[30\nname[Bob;age[25"

See implementations/python/sld.py

JavaScript/Node.js

npm install sld-format

const { decodeSLD, encodeSLD, decodeMLD, encodeMLD } = require('sld-format');

// Parse SLD
const data = decodeSLD("name[Alice;age[30~name[Bob;age[25~");
// Returns: [{name: "Alice", age: "30"}, {name: "Bob", age: "25"}]

// Parse MLD
const data = decodeMLD("name[Alice;age[30\nname[Bob;age[25");

// Generate SLD
const sld = encodeSLD([{name: "Alice", age: 30}, {name: "Bob", age: 25}]);

// Generate MLD
const mld = encodeMLD([{name: "Alice", age: 30}, {name: "Bob", age: 25}]);

See implementations/javascript/sld.js

Go

go get github.com/proteo5/sld-go

package main

import (
    "fmt"
    "github.com/proteo5/sld-go"
)

func main() {
    // Parse SLD
    data, err := sld.DecodeSLD("name[Alice;age[30~name[Bob;age[25~")
    
    // Parse MLD
    data, err := sld.DecodeMLD("name[Alice;age[30\nname[Bob;age[25")
    
    // Generate SLD
    sldStr, err := sld.EncodeSLD(data)
    
    // Generate MLD
    mldStr, err := sld.EncodeMLD(data)
}

See implementations/go/sld.go

C#

dotnet add package SLD.Format

using SLD;

// Parse SLD
var data = SLDParser.Decode("name[Alice;age[30~name[Bob;age[25~");

// Parse MLD
var data = SLDParser.DecodeMLD("name[Alice;age[30\nname[Bob;age[25");

// Generate SLD
string sld = SLDParser.EncodeSLD(data);

// Generate MLD
string mld = SLDParser.EncodeMLD(data);

See implementations/csharp/SLD.cs

PHP

composer require proteo5/sld-format

<?php
require 'vendor/autoload.php';

use SLD\Parser;

// Parse SLD
$data = Parser::decodeSLD("name[Alice;age[30~name[Bob;age[25~");

// Parse MLD
$data = Parser::decodeMLD("name[Alice;age[30\nname[Bob;age[25");

// Generate SLD
$sld = Parser::encodeSLD($data);

// Generate MLD
$mld = Parser::encodeMLD($data);
?>

See implementations/php/sld.php

Java

<dependency>
    <groupId>io.github.proteo5</groupId>
    <artifactId>sld-format</artifactId>
    <version>1.1.0</version>
</dependency>

import io.github.proteo5.sld.*;

// Parse SLD
List<Map<String, Object>> data = SLDParser.decode("name[Alice;age[30~name[Bob;age[25~");

// Parse MLD
List<Map<String, Object>> dataMLD = SLDParser.decodeMLD("name[Alice;age[30\nname[Bob;age[25");

// Generate SLD
String sld = SLDParser.encodeSLD(data);

// Generate MLD
String mld = SLDParser.encodeMLD(data);

See implementations/java/src/main/java/io/github/proteo5/sld/SLDParser.java

Unix Tool Examples (MLD)

MLD format is designed to work seamlessly with standard Unix tools:

grep - Filter Records

# Find all ERROR logs
grep "level\[ERROR" logs.mld

# Find users with admin role
grep "role\[admin" users.mld

# Find products out of stock
grep "inStock\[^0" products.mld

awk - Extract and Transform

# Extract all usernames
awk -F';' '{
  for(i=1;i<=NF;i++) {
    if($i ~ /^username\[/) {
      split($i,a,"["); print a[2]
    }
  }
}' users.mld

# Calculate average price
awk -F';' '
{
  for(i=1;i<=NF;i++) {
    if($i ~ /^price\[/) {
      split($i,a,"["); sum+=a[2]; count++
    }
  }
}
END {print "Average:", sum/count}' products.mld

sed - Edit Records

# Update all pending statuses to active
sed 's/status\[pending/status[active/g' orders.mld

# Remove verified field
sed 's/;verified\[[^;]*//g' users.mld

head/tail - Sample Data

# First 10 records
head -10 users.mld

# Last 5 log entries
tail -5 logs.mld

# Monitor logs in real-time
tail -f logs.mld

wc - Count Records

# Count total records
wc -l users.mld

# Count ERROR logs
grep "level\[ERROR" logs.mld | wc -l

Performance Benchmarks

Token Efficiency

Measured using GPT-4 tokenizer on identical dataset (100 user records):

Format	Total Tokens	Tokens/Record	vs JSON
SLD	2,200	22	-78% ✨
MLD	2,300	23	-77%
CSV	3,600	36	-71%
TOON	7,000	70	-44%
JSON (min)	8,500	85	-32%
JSON (fmt)	12,500	125	0%

Parsing Speed

Benchmark on 1M records (Python implementation):

Format	Parse Time	Generate Time	Memory
SLD	1.2s	0.8s	45 MB
MLD	1.3s	0.9s	47 MB
JSON	2.8s	1.9s	89 MB
CSV	0.9s	0.6s	38 MB

Note: CSV lacks nested structure support

File Size

Same 100 user records dataset:

Format	Size	Compression (gzip)
SLD	8.2 KB	2.1 KB
MLD	8.4 KB	2.2 KB
JSON	18.5 KB	4.8 KB
CSV	6.1 KB	1.9 KB

Security Considerations

Input Validation

Always validate and sanitize input before parsing:

# Python example
def safe_decode_sld(untrusted_input):
    # Validate length
    if len(untrusted_input) > MAX_INPUT_SIZE:
        raise ValueError("Input too large")
    
    # Validate characters
    if not all(ord(c) < 128 or c.isprintable() for c in untrusted_input):
        raise ValueError("Invalid characters")
    
    return decode_sld(untrusted_input)

Escape Injection Prevention

Never construct SLD/MLD strings with unescaped user input:

# WRONG - Vulnerable to injection
name = user_input  # Could contain ;~[{^
sld = f"name[{name};age[30~"

# CORRECT - Use encoder
from sld import escape_value
name = escape_value(user_input)  # Escapes special chars
sld = f"name[{name};age[30~"

# BEST - Use proper encoding function
data = {"name": user_input, "age": 30}
sld = encode_sld([data])

Size Limits

Implement reasonable limits:

MAX_INPUT_SIZE = 1_000_000  # 1MB
MAX_RECORDS = 10_000
MAX_FIELD_LENGTH = 10_000
MAX_NESTING_DEPTH = 10

Migration from v1.0

SLD v2.0 uses ; instead of | as field separator. See MIGRATION.md for complete guide.

Quick Migration

# Simple sed replacement (review output!)
sed 's/\^|/\x00/g; s/|/;/g; s/\x00/^;/g' old_v1.0.sld > new_v2.0.sld

Validation

# Verify migrated data
from sld import decode_sld
import json

# Original v1.0 data (using old parser)
original = decode_sld_v10(old_data)

# Migrated v2.0 data
migrated = decode_sld(new_data)

# Compare as JSON
assert json.dumps(original, sort_keys=True) == \
       json.dumps(migrated, sort_keys=True)

Contributing

We welcome contributions! See CONTRIBUTING.md for guidelines.

Development Setup

# Clone repository
git clone https://github.com/proteo5/sld.git
cd sld

# Install dependencies (Python example)
pip install -r requirements-dev.txt

# Run tests
pytest tests/

# Run linter
pylint sld/

Adding New Language Implementation

Create implementations/{language}/ directory
Implement encode_sld, decode_sld, encode_mld, decode_mld
Add comprehensive tests
Update implementations/README.md
Submit pull request

License

Apache License 2.0 - see LICENSE file for details.

Acknowledgments

Special thanks to @IceSolst, @travofoz, and @princessakano for giving the push needed to create this format.
Originally created as a humorous exploration of token efficiency
Created as both satire and serious optimization
Thanks to all contributors and early adopters

FAQ

Q: Should I use SLD or MLD?
A: Use SLD for network/compact storage, MLD for logs/streaming/Unix tools.

Q: Is v2.0 compatible with v1.0?
A: No. v2.0 uses ; instead of |. See MIGRATION.md.

Q: Can I mix SLD and MLD?
A: Yes! Convert with tr '~' '\n' (SLD→MLD) or tr '\n' '~' (MLD→SLD).

Q: How do I handle binary data?
A: Base64 encode first, then store as string value.

Q: What about Unicode?
A: Full UTF-8 support. All special chars can be escaped.

Q: Production ready?
A: Yes for v2.0 core. Well-tested, documented, multiple implementations.

Links

GitHub: github.com/proteo5/sld
Issues: github.com/proteo5/sld/issues
Discussions: github.com/proteo5/sld/discussions

Star ⭐ this repo if SLD/MLD saved you tokens!

🔄 Migration Guide

From JSON to SLD/MLD

Before (JSON):

{"id": 1, "name": "Alice", "tags": ["admin", "user"]}

After (SLD):

id[1;name[Alice;tags{admin~user}

After (MLD):

id[1;name[Alice;tags{admin~user}

From CSV to SLD/MLD

Before (CSV):

id,name,email
1,Alice,alice@example.com
2,Bob,bob@example.com

After (SLD):

1;Alice;alice@example.com~2;Bob;bob@example.com

After (MLD):

1;Alice;alice@example.com
2;Bob;bob@example.com

From v1.0 (Pipe) to v2.0 (Semicolon)

v1.0:

1|Alice|active

v2.0:

1;Alice;active

Conversion:

# Simple replacement (if no escaped pipes)
v11_data = v10_data.replace('|', ';')

# Proper conversion (handling escapes)
v11_data = convert_v10_to_v11(v10_data)

🤝 Contributing

We welcome contributions! See CONTRIBUTING.md for guidelines.

Name		Name	Last commit message	Last commit date
.github		.github
docs		docs
examples		examples
implementations		implementations
tests		tests
tools		tools
.editorconfig		.editorconfig
.gitignore		.gitignore
.markdownlint.json		.markdownlint.json
CHANGELOG.md		CHANGELOG.md
CODE_OF_CONDUCT.md		CODE_OF_CONDUCT.md
CONSISTENCY_REVIEW.md		CONSISTENCY_REVIEW.md
CONTRIBUTING.md		CONTRIBUTING.md
LICENSE		LICENSE
MIGRACION.md		MIGRACION.md
MIGRATION.md		MIGRATION.md
QUICK_REFERENCE.md		QUICK_REFERENCE.md
QUICK_REFERENCE_MLD.md		QUICK_REFERENCE_MLD.md
QUICK_REFERENCE_SLD.md		QUICK_REFERENCE_SLD.md
README.es.md		README.es.md
README.md		README.md
REFERENCIA_RAPIDA.md		REFERENCIA_RAPIDA.md
REFERENCIA_RAPIDA_MLD.md		REFERENCIA_RAPIDA_MLD.md
REFERENCIA_RAPIDA_SLD.md		REFERENCIA_RAPIDA_SLD.md
REGISTRO_CAMBIOS.md		REGISTRO_CAMBIOS.md
RFC_SLD_MLD.txt		RFC_SLD_MLD.txt
SECURITY.md		SECURITY.md
SLD.sln		SLD.sln
SPECIFICATION_MLD.md		SPECIFICATION_MLD.md
SPECIFICATION_SLD.md		SPECIFICATION_SLD.md
SYNTAX_GUIDE_MLD.md		SYNTAX_GUIDE_MLD.md
SYNTAX_GUIDE_SLD.md		SYNTAX_GUIDE_SLD.md

Latest commit

History

Folders and files

Repository files navigation

SLD/MLD - Single/Multi Line Data Format v2.0

Documentation

Core Documentation

v2.0 Optional Features

Language-Specific Documentation

Examples & Code

Validator (experimental)

Canonicalizer & Benchmark (experimental)

Format Converter (experimental)

Test Suite (comprehensive)

What is SLD/MLD?

Key Benefits

Token Comparison

Capabilities

Format Specifications

SLD Delimiters

MLD Delimiters

Escape Rules

When to Use SLD vs MLD

Use SLD When

Use MLD When

Format Conversion

Examples

Simple Objects

Products with Arrays

Application Logs (MLD Optimized)

Escaped Data

💻 Installation & Usage

Python

JavaScript/Node.js

Go

C#

PHP

Java

Unix Tool Examples (MLD)

grep - Filter Records

awk - Extract and Transform

sed - Edit Records

head/tail - Sample Data

wc - Count Records

Performance Benchmarks

Token Efficiency

Parsing Speed

File Size

Security Considerations

Input Validation

Escape Injection Prevention

Size Limits

Migration from v1.0

Quick Migration

Validation

Contributing

Development Setup

Adding New Language Implementation

License

Acknowledgments

FAQ

Links

🔄 Migration Guide

From JSON to SLD/MLD

From CSV to SLD/MLD

From v1.0 (Pipe) to v2.0 (Semicolon)

🤝 Contributing

About

Resources

Code of conduct

Contributing

Security policy

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages