ARK-OS/arkos/kernel/Documentation/driver-api/cxl/linux/example-configurations/multi-interleave.rst

.. SPDX-License-Identifier: GPL-2.0

======================
Multi-Level Interleave
======================
This cxl-cli configuration dump shows the following host configuration:

* A single socket system with one CXL root
* CXL Root has Four (4) CXL Host Bridges
* Two CXL Host Bridges have a two CXL Memory Expanders Attached each.
* The CXL root is configured to interleave across the two host bridges.
* Each host bridge with expanders interleaves across two endpoints.

This output is generated by :code:`cxl list -v` and describes the relationships
between objects exposed in :code:`/sys/bus/cxl/devices/`.

::

  [
    {
        "bus":"root0",
        "provider":"ACPI.CXL",
        "nr_dports":4,
        "dports":[
            {
                "dport":"pci0000:00",
                "alias":"ACPI0016:01",
                "id":0
            },
            {
                "dport":"pci0000:a8",
                "alias":"ACPI0016:02",
                "id":4
            },
            {
                "dport":"pci0000:2a",
                "alias":"ACPI0016:03",
                "id":1
            },
            {
                "dport":"pci0000:d2",
                "alias":"ACPI0016:00",
                "id":5
            }
        ],

This chunk shows the CXL "bus" (root0) has 4 downstream ports attached to CXL
Host Bridges.  The `Root` can be considered the singular upstream port attached
to the platform's memory controller - which routes memory requests to it.

The `ports:root0` section lays out how each of these downstream ports are
configured.  If a port is not configured (id's 0 and 1), they are omitted.

::

        "ports:root0":[
            {
                "port":"port1",
                "host":"pci0000:d2",
                "depth":1,
                "nr_dports":3,
                "dports":[
                    {
                        "dport":"0000:d2:01.1",
                        "alias":"device:02",
                        "id":0
                    },
                    {
                        "dport":"0000:d2:01.3",
                        "alias":"device:05",
                        "id":2
                    },
                    {
                        "dport":"0000:d2:07.1",
                        "alias":"device:0d",
                        "id":113
                    }
                ],

This chunk shows the available downstream ports associated with the CXL Host
Bridge :code:`port1`.  In this case, :code:`port1` has 3 available downstream
ports: :code:`dport0`, :code:`dport2`, and :code:`dport113`.

::

                "endpoints:port1":[
                    {
                        "endpoint":"endpoint5",
                        "host":"mem0",
                        "parent_dport":"0000:d2:01.1",
                        "depth":2,
                        "memdev":{
                            "memdev":"mem0",
                            "ram_size":137438953472,
                            "serial":0,
                            "numa_node":0,
                            "host":"0000:d3:00.0"
                        },
                        "decoders:endpoint5":[
                            {
                                "decoder":"decoder5.0",
                                "resource":825975898112,
                                "size":549755813888,
                                "interleave_ways":4,
                                "interleave_granularity":256,
                                "region":"region0",
                                "dpa_resource":0,
                                "dpa_size":137438953472,
                                "mode":"ram"
                            }
                        ]
                    },
                    {
                        "endpoint":"endpoint6",
                        "host":"mem1",
                        "parent_dport":"0000:d2:01.3",
                        "depth":2,
                        "memdev":{
                            "memdev":"mem1",
                            "ram_size":137438953472,
                            "serial":0,
                            "numa_node":0,
                            "host":"0000:d3:00.0"
                        },
                        "decoders:endpoint6":[
                            {
                                "decoder":"decoder6.0",
                                "resource":825975898112,
                                "size":549755813888,
                                "interleave_ways":4,
                                "interleave_granularity":256,
                                "region":"region0",
                                "dpa_resource":0,
                                "dpa_size":137438953472,
                                "mode":"ram"
                            }
                        ]
                    }
                ],

This chunk shows the endpoints attached to the host bridge :code:`port1`.

:code:`endpoint5` contains a single configured decoder :code:`decoder5.0`
which has the same interleave configuration as :code:`region0` (shown later).

:code:`endpoint6` contains a single configured decoder :code:`decoder5.0`
which has the same interleave configuration as :code:`region0` (shown later).

Next we have the decoders belonging to the host bridge:

::

                "decoders:port1":[
                    {
                        "decoder":"decoder1.0",
                        "resource":825975898112,
                        "size":549755813888,
                        "interleave_ways":2,
                        "interleave_granularity":512,
                        "region":"region0",
                        "nr_targets":2,
                        "targets":[
                            {
                                "target":"0000:d2:01.1",
                                "alias":"device:02",
                                "position":0,
                                "id":0
                            },
                            {
                                "target":"0000:d2:01.3",
                                "alias":"device:05",
                                "position":2,
                                "id":0
                            }
                        ]
                    }
                ]
            },

Host Bridge :code:`port1` has a single decoder (:code:`decoder1.0`), whose
targets are :code:`dport0` and :code:`dport2` - which are attached to
:code:`endpoint5` and :code:`endpoint6` respectively.

The following chunk shows a similar configuration for Host Bridge :code:`port3`,
the second host bridge with a memory device attached.

::

            {
                "port":"port3",
                "host":"pci0000:a8",
                "depth":1,
                "nr_dports":1,
                "dports":[
                    {
                        "dport":"0000:a8:01.1",
                        "alias":"device:c3",
                        "id":0
                    },
                    {
                        "dport":"0000:a8:01.3",
                        "alias":"device:c5",
                        "id":0
                    }
                ],
                "endpoints:port3":[
                    {
                        "endpoint":"endpoint7",
                        "host":"mem2",
                        "parent_dport":"0000:a8:01.1",
                        "depth":2,
                        "memdev":{
                            "memdev":"mem2",
                            "ram_size":137438953472,
                            "serial":0,
                            "numa_node":0,
                            "host":"0000:a9:00.0"
                        },
                        "decoders:endpoint7":[
                            {
                                "decoder":"decoder7.0",
                                "resource":825975898112,
                                "size":549755813888,
                                "interleave_ways":4,
                                "interleave_granularity":256,
                                "region":"region0",
                                "dpa_resource":0,
                                "dpa_size":137438953472,
                                "mode":"ram"
                            }
                        ]
                    },
                    {
                        "endpoint":"endpoint8",
                        "host":"mem3",
                        "parent_dport":"0000:a8:01.3",
                        "depth":2,
                        "memdev":{
                            "memdev":"mem3",
                            "ram_size":137438953472,
                            "serial":0,
                            "numa_node":0,
                            "host":"0000:a9:00.0"
                        },
                        "decoders:endpoint8":[
                            {
                                "decoder":"decoder8.0",
                                "resource":825975898112,
                                "size":549755813888,
                                "interleave_ways":4,
                                "interleave_granularity":256,
                                "region":"region0",
                                "dpa_resource":0,
                                "dpa_size":137438953472,
                                "mode":"ram"
                            }
                        ]
                    }
                ],
                "decoders:port3":[
                    {
                        "decoder":"decoder3.0",
                        "resource":825975898112,
                        "size":549755813888,
                        "interleave_ways":2,
                        "interleave_granularity":512,
                        "region":"region0",
                        "nr_targets":1,
                        "targets":[
                            {
                                "target":"0000:a8:01.1",
                                "alias":"device:c3",
                                "position":1,
                                "id":0
                            },
                            {
                                "target":"0000:a8:01.3",
                                "alias":"device:c5",
                                "position":3,
                                "id":0
                            }
                        ]
                    }
                ]
            },


The next chunk shows the two CXL host bridges without attached endpoints.

::

            {
                "port":"port2",
                "host":"pci0000:00",
                "depth":1,
                "nr_dports":2,
                "dports":[
                    {
                        "dport":"0000:00:01.3",
                        "alias":"device:55",
                        "id":2
                    },
                    {
                        "dport":"0000:00:07.1",
                        "alias":"device:5d",
                        "id":113
                    }
                ]
            },
            {
                "port":"port4",
                "host":"pci0000:2a",
                "depth":1,
                "nr_dports":1,
                "dports":[
                    {
                        "dport":"0000:2a:01.1",
                        "alias":"device:d0",
                        "id":0
                    }
                ]
            }
        ],

Next we have the `Root Decoders` belonging to :code:`root0`.  This root decoder
applies the interleave across the downstream ports :code:`port1` and
:code:`port3` - with a granularity of 256 bytes.

This information is generated by the CXL driver reading the ACPI CEDT CMFWS.

::

        "decoders:root0":[
            {
                "decoder":"decoder0.0",
                "resource":825975898112,
                "size":549755813888,
                "interleave_ways":2,
                "interleave_granularity":256,
                "max_available_extent":0,
                "volatile_capable":true,
                "nr_targets":2,
                "targets":[
                    {
                        "target":"pci0000:a8",
                        "alias":"ACPI0016:02",
                        "position":1,
                        "id":4
                    },
                    {
                        "target":"pci0000:d2",
                        "alias":"ACPI0016:00",
                        "position":0,
                        "id":5
                    }
                ],

Finally we have the `Memory Region` associated with the `Root Decoder`
:code:`decoder0.0`.  This region describes the overall interleave configuration
of the interleave set.  So we see there are a total of :code:`4` interleave
targets across 4 endpoint decoders.

::

                "regions:decoder0.0":[
                    {
                        "region":"region0",
                        "resource":825975898112,
                        "size":549755813888,
                        "type":"ram",
                        "interleave_ways":4,
                        "interleave_granularity":256,
                        "decode_state":"commit",
                        "mappings":[
                            {
                                "position":3,
                                "memdev":"mem3",
                                "decoder":"decoder8.0"
                            },
                            {
                                "position":2,
                                "memdev":"mem1",
                                "decoder":"decoder6.0"
                            }
                            {
                                "position":1,
                                "memdev":"mem2",
                                "decoder":"decoder7.0"
                            },
                            {
                                "position":0,
                                "memdev":"mem0",
                                "decoder":"decoder5.0"
                            }
                        ]
                    }
                ]
            }
        ]
    }
  ]