| 1 | # Netdata YAML Parser/Generator Module |
| 2 | |
| 3 | This module provides YAML parsing and generation capabilities for Netdata, with seamless conversion to/from json-c objects. It uses the libyaml library for parsing and generation, supporting the YAML subset that matches JSON 100%. |
| 4 | |
| 5 | ## Features |
| 6 | |
| 7 | ### Supported Operations |
| 8 | |
| 9 | 1. **Parse YAML from various sources:** |
| 10 | - String buffer (`yaml_parse_string`) |
| 11 | - File by filename (`yaml_parse_filename`) |
| 12 | - File descriptor (`yaml_parse_fd`) |
| 13 | |
| 14 | 2. **Generate YAML to various destinations:** |
| 15 | - BUFFER (`yaml_generate_to_buffer`) |
| 16 | - File by filename (`yaml_generate_to_filename`) |
| 17 | - File descriptor (`yaml_generate_to_fd`) |
| 18 | |
| 19 | 3. **Data types supported:** |
| 20 | - Null values |
| 21 | - Booleans |
| 22 | - Numbers (integers and floating-point) |
| 23 | - Strings |
| 24 | - Arrays |
| 25 | - Objects (maps) |
| 26 | |
| 27 | ## Supported YAML Features |
| 28 | |
| 29 | ### 1. Basic Data Types |
| 30 | |
| 31 | ```yaml |
| 32 | # Null values (all case-insensitive) |
| 33 | null_value1: null |
| 34 | null_value2: Null |
| 35 | null_value3: NULL |
| 36 | null_value4: ~ |
| 37 | |
| 38 | # Booleans (all case-insensitive) |
| 39 | bool_true: true # also: True, TRUE, yes, Yes, YES, on, On, ON |
| 40 | bool_false: false # also: False, FALSE, no, No, NO, off, Off, OFF |
| 41 | |
| 42 | # Numbers |
| 43 | integer: 42 |
| 44 | negative: -123 |
| 45 | float: 3.14159 |
| 46 | scientific: 1.23e-10 |
| 47 | ``` |
| 48 | |
| 49 | ### 2. Advanced Number Formats |
| 50 | |
| 51 | ```yaml |
| 52 | # Hexadecimal (parsed as integers) |
| 53 | hex_lower: 0x1a |
| 54 | hex_upper: 0XFF |
| 55 | hex_large: 0xDEADBEEF |
| 56 | |
| 57 | # Octal (YAML 1.2 style) |
| 58 | octal_new: 0o755 |
| 59 | octal_caps: 0O644 |
| 60 | |
| 61 | # Binary |
| 62 | binary_lower: 0b1010 |
| 63 | binary_upper: 0B11111111 |
| 64 | |
| 65 | # Numbers with underscores (YAML 1.2) |
| 66 | readable_int: 1_000_000 |
| 67 | readable_float: 3.141_592_653 |
| 68 | readable_hex: 0x1_A_B_C |
| 69 | readable_binary: 0b1010_1010 |
| 70 | ``` |
| 71 | |
| 72 | ### 3. Strings |
| 73 | |
| 74 | ```yaml |
| 75 | # Plain strings |
| 76 | plain: Hello World |
| 77 | |
| 78 | # Single quoted (literal strings) |
| 79 | single: 'This is a single quoted string' |
| 80 | single_escape: 'Can''t escape much in single quotes' |
| 81 | |
| 82 | # Double quoted (with escape sequences) |
| 83 | double: "Hello\nWorld" |
| 84 | escaped: "Tab:\t Quote:\" Backslash:\\" |
| 85 | |
| 86 | # Special strings that need quoting |
| 87 | quoted_null: "null" # Without quotes would be null value |
| 88 | quoted_bool: "true" # Without quotes would be boolean |
| 89 | quoted_number: "123" # Without quotes would be number |
| 90 | ``` |
| 91 | |
| 92 | ### 4. Collections |
| 93 | |
| 94 | ```yaml |
| 95 | # Arrays |
| 96 | simple_array: [1, 2, 3] |
| 97 | block_array: |
| 98 | - item1 |
| 99 | - item2 |
| 100 | - nested: value |
| 101 | |
| 102 | # Objects/Maps |
| 103 | simple_object: {key1: value1, key2: value2} |
| 104 | block_object: |
| 105 | name: John Doe |
| 106 | age: 30 |
| 107 | address: |
| 108 | street: 123 Main St |
| 109 | city: Anytown |
| 110 | ``` |
| 111 | |
| 112 | ### 5. Multiline Strings |
| 113 | |
| 114 | ```yaml |
| 115 | # Literal block scalar (preserves newlines and spacing) |
| 116 | literal: | |
| 117 | Line 1 |
| 118 | Line 2 |
| 119 | Indented line |
| 120 | |
| 121 | # Folded block scalar (folds newlines to spaces) |
| 122 | folded: > |
| 123 | This is a long |
| 124 | paragraph that will |
| 125 | be folded into a |
| 126 | single line. |
| 127 | ``` |
| 128 | |
| 129 | ## Round-Trip Consistency |
| 130 | |
| 131 | The module ensures round-trip consistency for most data types: |
| 132 | |
| 133 | ```c |
| 134 | // Original JSON |
| 135 | {"number": 1.0, "text": "hello", "flag": true} |
| 136 | |
| 137 | // Generated YAML |
| 138 | number: 1.0 |
| 139 | text: hello |
| 140 | flag: true |
| 141 | |
| 142 | // Parsed back to JSON |
| 143 | {"number": 1.0, "text": "hello", "flag": true} |
| 144 | ``` |
| 145 | |
| 146 | Special handling for floating-point numbers ensures that `1.0` remains `1.0` and doesn't become `1`. |
| 147 | |
| 148 | ## Limitations Due to libyaml |
| 149 | |
| 150 | ### 1. Unsupported Escape Sequences |
| 151 | |
| 152 | **Octal escapes** are not supported by libyaml: |
| 153 | ```yaml |
| 154 | # This will fail to parse |
| 155 | invalid: "\101" # Octal escape for 'A' |
| 156 | |
| 157 | # Workaround: use hex or unicode escapes |
| 158 | valid: "\x41" # Hex escape for 'A' |
| 159 | valid: "\u0041" # Unicode escape for 'A' |
| 160 | ``` |
| 161 | |
| 162 | ### 2. Null Bytes in Strings |
| 163 | |
| 164 | libyaml has issues with embedded null bytes: |
| 165 | ```yaml |
| 166 | # This may not work correctly |
| 167 | with_null: "before\x00after" |
| 168 | |
| 169 | # The null byte may terminate the string early |
| 170 | # or cause unexpected behavior |
| 171 | ``` |
| 172 | |
| 173 | ### 3. Single-Quoted Literal Newlines |
| 174 | |
| 175 | In single-quoted strings, literal newlines are converted to spaces by libyaml: |
| 176 | ```yaml |
| 177 | # Input |
| 178 | single: 'line1 |
| 179 | line2' |
| 180 | |
| 181 | # libyaml parses this as |
| 182 | single: 'line1 line2' |
| 183 | |
| 184 | # To preserve newlines, use double quotes or block scalars |
| 185 | double: "line1\nline2" |
| 186 | literal: | |
| 187 | line1 |
| 188 | line2 |
| 189 | ``` |
| 190 | |
| 191 | ### 4. Block Scalar Indentation |
| 192 | |
| 193 | Complex indentation in block scalars may not be preserved exactly: |
| 194 | ```yaml |
| 195 | # Deep indentation might be normalized |
| 196 | literal: | |
| 197 | deeply |
| 198 | indented |
| 199 | text |
| 200 | |
| 201 | # May lose some leading spaces depending on context |
| 202 | ``` |
| 203 | |
| 204 | ### 5. Invalid Syntax Detection |
| 205 | |
| 206 | Some invalid YAML syntax may be accepted by libyaml: |
| 207 | ```yaml |
| 208 | # This should be invalid but might parse |
| 209 | - item |
| 210 | - - invalid nesting |
| 211 | |
| 212 | # Proper nesting would be |
| 213 | - item |
| 214 | - |
| 215 | - valid nesting |
| 216 | ``` |
| 217 | |
| 218 | ## Usage Examples |
| 219 | |
| 220 | ### Parsing YAML |
| 221 | |
| 222 | ```c |
| 223 | // Parse from string |
| 224 | BUFFER *error = buffer_create(0, NULL); |
| 225 | const char *yaml = "name: Netdata\nversion: 1.0\n"; |
| 226 | struct json_object *json = yaml_parse_string(yaml, error, YAML2JSON_DEFAULT); |
| 227 | |
| 228 | if (buffer_strlen(error) > 0) { |
| 229 | fprintf(stderr, "Parse error: %s\n", buffer_tostring(error)); |
| 230 | } else { |
| 231 | // Use json object (note: NULL is valid for YAML null) |
| 232 | if (json) { |
| 233 | printf("Parsed: %s\n", json_object_to_json_string(json)); |
| 234 | json_object_put(json); |
| 235 | } else { |
| 236 | printf("Parsed YAML null value\n"); |
| 237 | } |
| 238 | } |
| 239 | |
| 240 | buffer_free(error); |
| 241 | ``` |
| 242 | |
| 243 | ### Generating YAML |
| 244 | |
| 245 | ```c |
| 246 | // Create JSON object |
| 247 | struct json_object *root = json_object_new_object(); |
| 248 | json_object_object_add(root, "name", json_object_new_string("Netdata")); |
| 249 | json_object_object_add(root, "version", json_object_new_double(1.0)); |
| 250 | |
| 251 | // Generate YAML |
| 252 | BUFFER *output = buffer_create(0, NULL); |
| 253 | BUFFER *error = buffer_create(0, NULL); |
| 254 | |
| 255 | if (yaml_generate_to_buffer(output, root, error)) { |
| 256 | printf("Generated YAML:\n%s", buffer_tostring(output)); |
| 257 | } else { |
| 258 | fprintf(stderr, "Generation error: %s\n", buffer_tostring(error)); |
| 259 | } |
| 260 | |
| 261 | json_object_put(root); |
| 262 | buffer_free(output); |
| 263 | buffer_free(error); |
| 264 | ``` |
| 265 | |
| 266 | ## Testing |
| 267 | |
| 268 | The module includes comprehensive unit tests covering: |
| 269 | |
| 270 | 1. **Basic tests** (`yaml-unittest.c`): |
| 271 | - All data types |
| 272 | - File operations |
| 273 | - Error handling |
| 274 | - Round-trip conversion |
| 275 | |
| 276 | 2. **Comprehensive tests** (`yaml-comprehensive-unittest.c`): |
| 277 | - All YAML string styles and number formats |
| 278 | - Edge cases, unicode and special characters |
| 279 | - Large documents and deep nesting |
| 280 | - Round-trip fidelity |
| 281 | |
| 282 | ## Implementation Notes |
| 283 | |
| 284 | 1. **Null Handling**: JSON null is represented as C NULL pointer in json-c, which is properly handled in both parsing and generation. |
| 285 | |
| 286 | 2. **Number Parsing**: The parser attempts to identify number types in this order: |
| 287 | - Hexadecimal (0x/0X prefix) |
| 288 | - Octal (0o/0O prefix) |
| 289 | - Binary (0b/0B prefix) |
| 290 | - Integer (decimal) |
| 291 | - Floating-point |
| 292 | |
| 293 | 3. **String Quoting**: Strings are automatically quoted during generation if they: |
| 294 | - Could be misinterpreted as null, boolean, or number |
| 295 | - Contain special characters |
| 296 | - Have leading/trailing spaces |
| 297 | - Contain newlines |
| 298 | |
| 299 | 4. **Memory Management**: All allocated memory is properly freed. The module has been tested with AddressSanitizer and LeakSanitizer. |
| 300 | |
| 301 | ## Thread Safety |
| 302 | |
| 303 | The module is thread-safe as long as: |
| 304 | - Different threads use different parser/emitter instances |
| 305 | - json-c objects are not shared between threads without proper synchronization |
| 306 | - BUFFER objects are not shared between threads without proper synchronization |