YANG adventures in NetGauze: YANG Schema AST, Schema Tree, Data Tree: What the heck?!
NetGauze has been my ongoing side project for a few years to provide the building blocks for building network management applications. It supports BGP, BMP, Netflow V9, and IPFIX protocols. The next frontier for NetGauze is to understand YANG and be able to communicate to the network via NETCONF and RESTCONF. I’ve been studying how YANG works and decided to share my findings in several posts.
I assume the user is very familiar with YANG, and I aim to focus on some of the non-obvious details rather than a YANG tutorial.
In this article, I review the foundational concepts needed to implement any YANG parser. I start with a simple example of a YANG module to make the discussion more applicable. Then, I show examples of four main concepts: Parsed YANG Schema Abstract Syntax Tree (AST), Resolved YANG Schema AST, Schma Tree, and Data Tree.
Motivating Example
To drive the discussion with a concrete example, let’s define three simple YANG modules: M1, M2, and M3. M1 imports M2 to use a grouping defined in M1. M3 augments M1.
module M1 {
yang-version 1.1;
namespace "urn:ps:hassany:yang:examples:M1";
prefix m1;
import M2 {
prefix m2;
}
contact
"Author: Ahmed Elhassany
<a.hassany@gmail.com>
";
description
"Example test module";
revision 2024-02-10 {
description
"Initial revision.";
}
typedef IfindexType {
type uint16;
description
"Interface Index";
}
container interface {
leaf version {
type uint8;
mandatory true;
}
leaf node {
type string;
mandatory true;
}
leaf ifindex {
type IfindexType;
mandatory true;
}
uses m2:IfCounters {
refine "counters" {
must "../version = 1 or not(../version)" {
error-message
"Version must be 1 or undefined";
}
}
}
anydata L4;
anyxml L5;
}
notification updates {
leaf node {
type leafref {
path "/interface/node";
}
}
leaf ifindex {
type leafref {
path "/interface/ifindex";
}
}
container counters {
leaf in_pkts {
type leafref {
path "/interface/counters/in_pkts";
}
}
leaf out_pkts {
type leafref {
path "/interface/counters/out_pkts";
}
}
}
}
}module M2 {
yang-version 1.1;
namespace "urn:ps:hassany:yang:examples:M2";
prefix m2;contact
"Author: Ahmed Elhassany
<a.hassany@gmail.com>
";
description
"Example test module";revision 2024-02-10 {
description
"Initial revision.";
}grouping IfCounters {
container counters {
leaf in_pkts {
type uint32;
}
leaf out_pkts {
type uint32;
}
}
}
}
module M3 {
yang-version 1.1;
namespace "urn:ps:hassany:yang:examples:M3";
prefix m3;
import M1 {
prefix m1;
}
contact
"Author: Ahmed Elhassany
<a.hassany@gmail.com>
";
description
"Example test module";
revision 2024-02-10 {
description
"Initial revision.";
}
augment "/m1:interface" {
leaf ip_address {
type uint32;
}
}
}And an example YANG data:
<interface xmlns="urn:ps:hassany:yang:examples:M1">
<version>1</version>
<node>node1</node>
<ifindex>2</ifindex>
<counters>
<in_pkts>1</in_pkts>
<out_pkts>2</out_pkts>
</counters>
<L4>
<example>Some unspecified YANG data</example>
</L4>
<L5>
<example>Some unspecified XML data</example>
</L5>
<ip_address xmlns="urn:ps:hassany:yang:examples:M3">1232</ip_address>
</interface>YANG Tree Diagrams
Since YANG Tree Diagrams [RFC8340] are commonly used tools to describe YANG modules, I have included them for M1, M2, and M3. However, caution, YANG Tree Diagrams are intended to only show “data model”.
module: M1
+--rw C1
+--rw L1? uint8
+--rw L2? T1
+--rw M2L1? uint32
+--rw L4? anydata
+--rw L5? anyxmlNote M2 does not define a data model, only a grouping, hence the tree diagram is empty.
module: M2module: M3
augment /m1:interface:
+--rw ip_address? uint32(Parsed) YANG Schema Abstract Syntax Tree (AST)
YANG schema parser processing M1, M2, and M3 will produce an AST for each module. This AST will follow the ABNF grammar defined in RFC7950 Section 14. A group of YANG Schema ASTs forms a YANG Schema context. At this stage, the parser can apply some checks to validate that the YANG schema file is valid. However, not all aspects are validated. For instance, it can validate that the augment statement in M3 is syntactically correct but doesn’t yet evaluate if the augmented path exists in M1. Note: libyang calls the AST parsed schema.
A YANG Schema AST by itself is not usable for parsing YANG data since many of the references still need to be solved (e.g., uses statements, leaf using v from imported module, and augmentation). While someone might attempt to write a YANG parser using only parsed schemas, the parser code will be very complicated since it must constantly jump around many YANG Schema ASTs while parsing each YANG data file.
To illustrate the point, the following are incomplete ASTs for M1, M2, and M3. I only highlight the important nodes in the tree. Note, in the AST for M3, we have a node that augments a schema node id in M1, but the tree doesn’t reference M1 at this point. Also, note in M1 leaf ifindex has a reference to type
Resolved (Compiled) YANG Schema AST
A better approach than just Parsed YANG Schema ASTs is to pre-resolve all the references and produce a single YANG schema tree that fully specifies all branches. RFC7950 does not specify this tree and left it as an implementation detail as far as I can tell. libyang refers to this resolved tree as a compiled yang schema tree. pyang doesn’t state the term “compiled schema tree” in its codebase, but pyang also follow a similar approach to libyang; see pyang parser. I, personally, prefer the term Resolved YANG Schema AST since, in computer science, the term “compilation” has been commonly used to refer to the process of converting high-level code into low-level code.
To illustrate how Resolved YANG Schema ASTs work, I illustrate the resolved AST for M3. Note blue leaves are the ones who got resolved.
But RFC7950 defines Schema Tree and Schema Node: What do they mean?
Let’s start with some definitions from RFC7950 to establish some common grounds:
- schema node: [RFC7950] A node in the schema tree. One of
action,container,leaf,leaf-list,list,choice,case,rpc,input,output,notification,anydata, andanyxml. - schema tree: [RFC7950] The definition hierarchy specified within a module.
- module: [RFC7950] A YANG module defines hierarchies of schema nodes. With its definitions and the definitions it imports or includes from elsewhere, a module is self-contained and “compilable”.
By examining these definitions, Schema Tree is a subset of YANG Schema AST capable of carrying YANG data. But it suffers from the same limitation of the Parsed YANG Schema AST, in the sense that only some leaves are fully specified. A resolution process is needed to make the Schema Tree useful for parsing and validating YANG Data.
I show only an example Schema Tree for only M1 since Schema Tree is a subset of the ASTs we illustrated before.
Data Tree
While Schema Tree (compiled or parsed) defined the domain of valid YANG Data (set of all possible valid input data) accepted by the schema, a Data Tree is a specific instantiation of the data nodes of YANG data.
The data tree of our example is as follow (note I have annotated the values inside each leaf):
Where to next?
YANG tools have devised different terms and ways to resolve the schema tree into a usable form (note YANG RFC7950 Section 5.5 defined some methods to resolve groupings). That caused many misconceptions and misunderstandings at least in my head since the terms and methods were not well defined. A better way forward is to define these steps and agree on common terminology.
References
- NetGauze https://github.com/NetGauze/NetGauze
- RFC7950 The YANG 1.1 Data Modeling Language: https://datatracker.ietf.org/doc/html/rfc7950
- libyang: https://github.com/CESNET/libyang
- pyang: https://github.com/mbj4668/pyang
- RFC 8340 YANG Tree Diagrams https://datatracker.ietf.org/doc/html/rfc8340
