Recursive AI

Edit UTF-16 YAML without changing its encoding in Go

oneyaml-go is a free, open-source YAML library for Go from Recursive AI. MIT or Apache-2.0, no dependencies.

Edit UTF-16 big-endian bytes without converting them to UTF-8. If an edit replaces the ASCII start with a non-ASCII character, it adds a byte order mark (BOM).

Terminal
go get oneyaml.dev/go@v1.0.0

Requires Go 1.24 or later. Run go get in a Go module.

GO
package main
 
import (
    "fmt"
    "os"
    yaml "oneyaml.dev/go"
)
 
func main() {
    // UTF-16BE for a: 1\nb: 2, no BOM.
    input := []byte("\x00a\x00:\x00 \x001" +
        "\x00\n\x00b\x00:\x00 \x002")
    doc, err := yaml.Parse(input)
    if err != nil { panic(err) }
    // JSON Pointer: /a selects the key;
    // "" selects the whole document.
    if err := doc.Set("/a", 2); err != nil {
        panic(err)
    }
    if err := os.WriteFile("edited.yaml",
        doc.Bytes(), 0600); err != nil {
        panic(err)
    }
    saved, err := os.ReadFile("edited.yaml")
    if err != nil { panic(err) }
    var values map[string]int
    if err := yaml.Unmarshal(saved,
        &values); err != nil {
        panic(err)
    }
    fmt.Printf("Start: % x\n", saved[:2])
    fmt.Println("a:", values["a"])
}

Unzip the example, then go run ..

Writes edited.yaml and prints:

Text
Start: 00 61
a: 2

The full program

The download builds UTF-16BE bytes from a: 1\nb: 2 with Go’s unicode/utf16 and encoding/binary packages. It prints the bytes before and after editing, then reads the saved file as YAML.

GO
package main
 
import (
	"encoding/binary"
	"fmt"
	"log"
	"os"
	"unicode/utf16"
 
	yaml "oneyaml.dev/go"
)
 
func main() {
	// The input starts with ASCII 'a' and has no byte order mark.
	var input []byte
	for _, unit := range utf16.Encode([]rune("a: 1\nb: 2")) {
		input = binary.BigEndian.AppendUint16(input, unit)
	}
	doc, err := yaml.Parse(input)
	if err != nil {
		log.Fatal(err)
	}
	if err := doc.Set("/a", 2); err != nil {
		log.Fatal(err)
	}
	const path = "edited.yaml"
	if err := os.WriteFile(path, doc.Bytes(), 0600); err != nil {
		log.Fatal(err)
	}
	saved, err := os.ReadFile(path)
	if err != nil {
		log.Fatal(err)
	}
	var values map[string]int
	if err := yaml.Unmarshal(saved, &values); err != nil {
		log.Fatal(err)
	}
	fmt.Printf("Before: % x\nAfter:  % x\n", input, saved)
	fmt.Printf("Read back: a=%d, b=%d\n", values["a"], values["b"])
}

go run . prints:

Text
Before: 00 61 00 3a 00 20 00 31 00 0a 00 62 00 3a 00 20 00 32
After:  00 61 00 3a 00 20 00 32 00 0a 00 62 00 3a 00 20 00 32
Read back: a=2, b=2

Only the byte for the digit in a changes. If Parse or Set returns an error, the program stops before writing the output file.

Variations

Replace the root with text that doesn’t start with ASCII. On a freshly parsed doc from the example, pass the empty pointer to replace the whole YAML value. Decode the result into a string instead of the map in the full program.

GO
if err := doc.Set("", "日本語"); err != nil {
    panic(err)
}
var text string
if err := yaml.Unmarshal(doc.Bytes(), &text); err != nil {
    panic(err)
}
fmt.Println(text)

Prints 日本語. This edit adds a BOM so the result is still recognizable as UTF-16BE.

The short answer

Pass the original bytes to oneyaml.Parse, change the value with doc.Set, then save doc.Bytes(). In this example, /a selects the key a at the top of the mapping. The new file is still UTF-16BE, starts with the same ASCII key, and doesn’t gain a byte order mark. Read the bytes back with os.ReadFile and decode them to check the result.

What encoding does YAML use?

A YAML file can be UTF-8 rather than UTF-16; don’t infer its encoding from the .yaml extension. The YAML specification’s character-encoding rules describe how a reader distinguishes the encodings from the first bytes. Here 00 61 is the UTF-16BE encoding of a: the high byte comes first. In UTF-8 the same ASCII letter would occupy just one byte, 61. Similarly, the newline in the example is 00 0a, not a lone 0a.

If a file is already on disk, pass the result of os.ReadFile to Parse. The conversion loop in the download only creates the example input. You don’t need to decode the file to runes before parsing it, and you shouldn’t rebuild the file from a decoded Go map when preserving its original encoding is the goal. The document API tracks the encoding of the input it actually receives.

Does UTF-16 YAML need a byte order mark?

Not for the example’s ASCII start. The first character is a, so the first two bytes let the reader distinguish big-endian UTF-16 from little-endian UTF-16. But the question changes if you replace the start of the file. In oneyaml-go, the marked and unmarked outcomes of these edits depend on what the edited text starts with. Replacing the entire mapping with 日本語 keeps UTF-16BE but adds a mark. Updating /a leaves the ASCII a at the start and doesn’t add one.

Preserving an encoding and preserving the exact bytes at the start of a file are separate requirements. If another program consumes the edited file, check that it accepts the resulting encoding and BOM. In the download, saving to a separate path leaves your input alone while you inspect the saved bytes or open the result in that program.

All oneyaml Go recipes