Description of the Artifact
The authors provide a link to a github repository
(https://github.com/demikernel/demikernel/) that contains source for
catnip-libos as well as associated dependencies.
catnip-libos consists of four
Rust source files covering DPDK, memory, a runtime, and a library interface for
C code. It also includes source for two example echo applications (
udp.rs) corresponding to the Cantip (TCP) and Catnip (UDP)
microbenchmarks as described in the paper.
The authors also pointed the reviewers to another version of interest within
the demikernel repository: the
osdi-legacy-code branch, which points to a
version of the code before significant re-factoring and re-implementation.
This contains other libOSes, in particular Catnap and Catmint.
The demikernel repository contains a README that covers how to build the contents of the repository as well as a statement about contributing to the project. It is licensed under the MIT license.
The authors also provided a link to a repository that contains an echo
benchmark (https://github.com/demikernel/echo) . This contains code
corresponding to the Linux/POSIX echo microbenchmark described in the paper.
This repo provides a
Makefile for building the contents.
In addition to third-party libraries such as DPDK and SPDK, the artifact makes use of other repositories developed by the authors and their collaborators, notably https://github.com/demikernel/catnip and https://github.com/demikernel/dpdk-rs .
Environment(s) Used for Testing
The evaluators tried the artifact in two environments:
In CloudLab, using two
c6525-100gnodes. These machines have AMD EPYC Rome 24 core processors, 128GB of RAM, two PCIe v4.0 NVMe drives (NAND flash, unlike the Optane used for the authors’ own experiments), and a ConnectX-5 Ex NIC with two 100Gbps ports. Evaluators used the CloudLab Ubuntu 18.04 image, as that was the OS recommended by the authors: specifically, Ubuntu 18.04.1 with kernel
A set of author-provided VMs on Azure. Each evaluator was provided access to two VMs which contain 4 CPUs whose model name is
Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30GHz, and a Mellanox Ethernet controller with type
The authors also provided a link to a public CloudLab profile (https://www.cloudlab.us/p/Demeter/testing-pair/9) that allocates nodes with 25 Gbps Mellanox NICs.
Evaluators used the Mellanox OFED
MLNX_OFED-5.4-126.96.36.199 from http://www.mellanox.com/downloads/ofed/MLNX_OFED-5.4-188.8.131.52/MLNX_OFED_LINUX-5.4-184.108.40.206-ubuntu18.04-x86_64.tgz
Step-By-Step Instructions to Exercise the Artifact
Hardware allocation on CloudLab
Allocate an experiment on CloudLab using the
small-lan profile. The version
specifically used by the AEC was version 28:
Request two nodes, the Ubuntu 18.04 image, and hardware type
Under “Advanced Options”, set the link speed to 100Gbps, set the “Temp
Filesystem Max Space” checkbox, and set “Temporary Filesystem Mount Point” to
“/mydata”. The filesystem-related options give sufficient space for
all files required.
Once the CloudLab machines boot, run
sudo chown $USER /mydata
Installation of mlx5
This step and may not need to be performed on machines that already have the OFED installed. (For example, it is already installed in the authors’ CloudLab profiles.) Perform the following on both machines in the experiment.
wget http://www.mellanox.com/downloads/ofed/MLNX_OFED-5.4-220.127.116.11/MLNX_OFED_LINUX-5.4-18.104.22.168-ubuntu18.04-x86_64.tgz tar -xvf MLNX_OFED_LINUX-5.4-22.214.171.124-ubuntu18.04-x86_64.tgz cd MLNX_OFED_LINUX-5.4-126.96.36.199-ubuntu18.04-x86_64/ sudo ./mlnxofedinstall --upstream-libs --dpdk
Do you want to continue?[y/N]:, answer
The install script adds a few packages that were not previously installed on my system. The script did not update the NIC firmware, saying:
The firmware for this device is not distributed inside Mellanox driver: 41:00.0 (PSID: DEL0000000004) To obtain firmware for this device, please contact your HW vendor.
This did not seem to present a problem.
When the process finishes, it prompts the user to run
to load the driver - however, rebooting the machines (
sudo shutdown -r now)
may be more reliable, because reloading the driver may interrupt the
ssh session and prevent the driver reload from working.
When the machines finish rebooting from installing the OFED, follow the
https://github.com/demikernel/demikernel/blob/master/README.md (the AEC used
Installing demikernel repo and dependencies
export WORKDIR=/mydata cd $WORKDIR git clone --recursive https://github.com/demikernel/demikernel.git cd $WORKDIR/demikernel sudo -H scripts/setup/debian.sh curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh source $HOME/.cargo/env make dpdk
In the stage for installing
rustup, take the default options when prompted.
(Note, this step is not required on the authors’ profiles, which have rust
make of DPDK succeeded for the evaluators, with the following information
about what was included, and not included, in the build:
Message: ================= Libraries Enabled ================= libs: kvargs, telemetry, eal, ring, rcu, mempool, mbuf, net, meter, ethdev, pci, cmdline, metrics, hash, timer, acl, bbdev, bitratestats, cfgfile, compressdev, cryptodev, distributor, efd, eventdev, gro, gso, ip_frag, jobstats, kni, latencystats, lpm, member, power, pdump, rawdev, regexdev, rib, reorder, sched, security, stack, vhost, ipsec, fib, port, table, pipeline, flow_classify, bpf, graph, node, Message: =============== Drivers Enabled =============== common: cpt, dpaax, iavf, octeontx, octeontx2, sfc_efx, mlx5, qat, bus: dpaa, fslmc, ifpga, pci, vdev, vmbus, mempool: bucket, dpaa, dpaa2, octeontx, octeontx2, ring, stack, net: af_packet, ark, atlantic, avp, axgbe, bond, bnx2x, bnxt, cxgbe, dpaa, dpaa2, e1000, ena, enetc, enic, failsafe, fm10k, i40e, hinic, hns3, iavf, ice, igc, ionic, ixgbe, kni, liquidio, memif, mlx5, netvsc, nfp, null, octeontx, octeontx2, octeontx_ep, pcap, pfe, qede, ring, sfc, softnic, tap, thunderx, txgbe, vdev_netvsc, vhost, virtio, vmxnet3, raw: dpaa2_cmdif, dpaa2_qdma, ioat, ntb, octeontx2_dma, octeontx2_ep, skeleton, crypto: bcmfs, caam_jr, ccp, dpaa_sec, dpaa2_sec, nitrox, null, octeontx, octeontx2, openssl, scheduler, virtio, compress: mlx5, octeontx, zlib, regex: mlx5, octeontx2, vdpa: ifc, mlx5, event: dlb, dlb2, dpaa, dpaa2, octeontx2, opdl, skeleton, sw, dsw, octeontx, baseband: null, turbo_sw, fpga_lte_fec, fpga_5gnr_fec, acc100, Message: ================= Content Skipped ================= libs: drivers: common/mvep: missing dependency, "libmusdk" net/af_xdp: missing dependency, "libbpf" net/ipn3ke: missing dependency, "libfdt" net/mlx4: missing dependency, "mlx4" net/mvneta: missing dependency, "libmusdk" net/mvpp2: missing dependency, "libmusdk" net/nfb: missing dependency, "libnfb" net/szedata2: missing dependency, "libsze2" raw/ifpga: missing dependency, "libfdt" crypto/aesni_gcm: missing dependency, "libIPSec_MB" crypto/aesni_mb: missing dependency, "libIPSec_MB" crypto/armv8: missing dependency, "libAArch64crypto" crypto/kasumi: missing dependency, "libIPSec_MB" crypto/mvsam: missing dependency, "libmusdk" crypto/snow3g: missing dependency, "libIPSec_MB" crypto/zuc: missing dependency, "libIPSec_MB" compress/isal: missing dependency, "libisal"
DPDK installs itself into various directories in
$HOME, such as
This step, documented in the README, is straightforward:
cd $WORKDIR/demikernel make
Running demikernel programs (catnip (UDP) and catnip (TCP))
Building the catnip programs
These programs are not built by default, but building them is straightforward
cd $WORKDIR/demikernel make demikernel-examples
This step places executables in
Setting up hugepages
Hugepages must be set up in order to run the DPDK code. The
script provided in the demikernel repository at the time of evaluation
scripts/setup/hugepages.sh) did not work. An evaluator created the
following script to replace it:
#!/bin/sh echo 1024 | sudo tee /sys/devices/system/node/node*/hugepages/hugepages-2048kB/nr_hugepages echo 64 | sudo tee /sys/devices/system/node/node*/hugepages/hugepages-1048576kB/nr_hugepages mkdir -p /mnt/huge || true mkdir -p /mnt/huge/2MB || true mkdir -p /mnt/huge/1GB || true mount -t hugetlbfs -o pagesize=2M nodev /mnt/huge/2MB mount -t hugetlbfs -o pagesize=1G nodev /mnt/huge/1GB
Replace the contents of
scripts/setup/hugepages.sh with this, and execute it:
cd $WORKDIR/demikernel sudo scripts/setup/hugepages.sh
This needs to be done on both machines. If nodes get rebooted, this must be run after every boot.
Creating configuration files
The catnip programs require configuration files: modify
scripts/config/default.yaml on each of the machines. Configuration files used
by one of the evaluators on CloudLab are below; note, however, that they are
specific to the machines used and may need to be modified to run elsewhere,
even on other CloudLab machines. You can find the nodes’ IP addresses with
ifconfig and the PCI bus addresses (for the
eal_init line) for the cards
lspci | grep Mellanox . On CloudLab, make sure to look for interfaces
with private IP addresses, as these are on the “experiment” network.
client: connect_to: host: 10.10.1.2 port: 12345 client: host: 10.10.1.1 port: 12345 server: bind: host: 10.10.1.2 port: 12345 client: host: 10.10.1.1 port: 12345 catnip: my_ipv4_addr: 10.10.1.1 arp_disable: true dpdk: eal_init: ["-c", "0xff", "-n", "4", "-w", "41:00.0","--proc-type=auto"]
client: connect_to: host: 10.10.1.2 port: 12345 client: host: 10.10.1.1 port: 12345 server: bind: host: 10.10.1.2 port: 12345 client: host: 10.10.1.1 port: 12345 catnip: my_ipv4_addr: 10.10.1.2 arp_disable: true dpdk: eal_init: ["-c", "0xff", "-n", "4", "-w", "41:00.0","--proc-type=auto"]
Running catnip (UDP)
The catnip example programs
tcp respectively get built in
In addition to the configuration files above, they require a number of
environment variables. Additionally, because DPDK installs itself in
it is necessary to set
In addition to the configuration files above, they require a number of environment variables, which were provided via comments by the authors. Additionally, because DPDK installs itself in $HOME/, it is necessary to set LD_LIBRARY_PATH=$HOME/lib/x86_64-linux-gnu.
On the machine you are using as the server (
node1 in these examples), run:
cd $WORKDIR/demikernel sudo env LD_LIBRARY_PATH=$HOME/lib/x86_64-linux-gnu/ MSS=9000 MTU=1500 NUM_ITERS=1000 BUFFER_SIZE=64 ECHO_SERVER=yes src/target/release/examples/udp scripts/config/default.yaml
On the client (
cd $WORKDIR/demikernel sudo env LD_LIBRARY_PATH=$HOME/lib/x86_64-linux-gnu/ MSS=9000 MTU=1500 NUM_ITERS=1000 BUFFER_SIZE=64 ECHO_CLIENT=yes src/target/release/examples/udp scripts/config/default.yaml
The evaluators were able to get results that were not identical to those in the paper (Figure 5), but as close as can be expected given that they were run on a different hardware platform than used for the paper.
Running catnip (TCP)
Running the TCP version of this microbenchmark is essentially the same as the UDP one.
On the machine being used as the server (
node1 in these examples), run:
cd $WORKDIR/demikernel sudo env LD_LIBRARY_PATH=$HOME/lib/x86_64-linux-gnu/ MSS=9000 MTU=1500 NUM_ITERS=1000 BUFFER_SIZE=64 ECHO_SERVER=yes src/target/release/examples/tcp scripts/config/default.yaml
On the client (
cd $WORKDIR/demikernel sudo env LD_LIBRARY_PATH=$HOME/lib/x86_64-linux-gnu/ MSS=9000 MTU=1500 NUM_ITERS=1000 BUFFER_SIZE=64 ECHO_CLIENT=yes src/target/release/examples/tcp scripts/config/default.yaml
The evaluators got latency numbers very close to the values found for UDP on the same machines.
Building and running programs from the echo repository: Linux/POSIX
The baseline Linux/POSIX microbenchmark can be found in a repository at https://github.com/demikernel/echo
Building programs from echo repository
Do the following to install the echo repository, as documented in the README:
cd $WORKDIR git clone https://github.com/demikernel/echo.git
$WORKDIR was still set to
/mydata as above)
Install build dependencies:
sudo apt-get install libyaml-cpp-dev libboost-all-dev
At this point, the build should succeed for both the
cd $WORKDIR/echo make
Running POSIX programs from the echo repository (Linux/POSIX)
The programs under
src/posix test the configuration called
POSIX/Linux in the paper. On the server machine (
node1 in the CloudLab experiment), run:
cd $WORKDIR/echo ./bin/server
… and on the other (
cd $WORKDIR/echo ./bin/client --ip 10.10.1.2
Building and running programs from the
osdi-legacy-code branch of
demikernel (catnap and catmint)
Some of the microbenchmarks in the paper can be found in the
osdi-legacy-code branch of the main demikernel repository.
Checking out the
osdi-legacy-code branch contains references to four broken submodules, all related to the REDIS implementation. A reliable method to obtain a copy of the
demikernel repository with the
osdi-legacy-code branch is the following:
cd $WORKDIR git clone --branch osdi-legacy-code https://github.com/demikernel/demikernel.git osdi-legacy-code cd osdi-legacy-code git submodule init git submodule deinit -f -- submodules/redis-dpdk-catnip/ git rm -f submodules/redis-dpdk-catnip/ git submodule deinit -f -- submodules/redis-lwip git rm -f submodules/redis-lwip git submodule deinit -f -- submodules/redis-posix git rm -f submodules/redis-posix git submodule deinit -f -- submodules/redis-rdma git rm -f submodules/redis-rdma git add .gitmodules submodules/ git commit -m "Removed broken Redis submodules" git submodule update
Building programs in the
This is straighforward, following instructions from the
BUILDING.md file in this branch. The basic steps are:
Install additional dependencies:
cd $WORKDIR/osdi-legacy-code sudo scripts/setup/debian.sh
Set up and configure build environment:
cd $WORKDIR/osdi-legacy-code mkdir -p build/release cd build/release cmake $WORKDIR/osdi-legacy-code ccmake $WORKDIR/osdi-legacy-code
ccmake step, change
DPDK_MLX5_SUPPORT to ON; “cg” saves, exits, and reconfigures.
Only certain targets build successfully, so these are the ones to build to run the appropriate programs for this section:
cd $WORKDIR/osdi-legacy-code/build/release make dmtr-rdma-echo dmtr-posix-echo
On the server (
cd $WORKDIR/osdi-legacy-code/build/release ./src/c++/apps/echo/dmtr-posix-server --ip 10.10.1.2
On the client (
cd $WORKDIR/osdi-legacy-code/build/release ./src/c++/apps/echo/dmtr-posix-client --ip 10.10.1.2 -i 10000
On the server (
cd $WORKDIR/osdi-legacy-code/build/release ./src/c++/apps/echo/dmtr-rdma-server --ip 10.10.1.2
On the client (
cd $WORKDIR/osdi-legacy-code/build/release ./src/c++/apps/echo/dmtr-rdma-client --ip 10.10.1.2 -i 10000
How The Artifact Supports The Paper
echo repositories make available four of the libOSes described in the paper. Excercising them requires the documentation in this summary.
Additional Notes and Resources
In addition to the
master branch in the
demikernel repository, the
osdi-legacy-code branch is likely to be of interest to others interested in this artifact. The authors stated that a re-write of demikernel was in progress at the time of the review, and more may be available in the
main branch in the future.