Debugging
Since Youki uses pipe and double-fork in the creating phase, it is hard to debug what happened. You might encounter the error message, "Broken pipe ..." Unfortunately, this error message has only information that a child process exists with an error for some reason.
This section will give some tips to debug youki to know what happens in the child processes.
bpftrace
bpftrace is an eBPF based tool. In the case of youki, you can catch the system calls youki issued.
For example, if you catch write system calls, you can see the log output until the middle of process. It allows you to do something similar to print debugging.
How to debug
-
You need to install bpftrace, please refer to the official documentation to know how to install it.
-
Before running the process or comannd you want to debug, run the following command in another terminal.
You need the root privilege to run it.
$ cd ${youki_repo} $ just hack-bpftrace -
Run the command you want to debug.
For example
-
Run the bpftrace script.
$ just hack-bpftrace BPFTRACE_STRLEN=120 ./hack/debug.bt Attaching 13 probes... Tracing Youki syscalls... Hit Ctrl-C to end. TIME COMMAND PID EVENT CONTENT -
Run the Kubernetes cluster using kind with youki
$ cd ${youki_repo} $ just test-kind docker buildx build --output=bin/ -f tests/k8s/Dockerfile --target kind-bin . ... Creating cluster "youki" ... ... kubectl --context=kind-youki apply -f tests/k8s/deploy.yaml runtimeclass.node.k8s.io/youki created deployment.apps/nginx-deployment created ... kubectl --context=kind-youki delete -f tests/k8s/deploy.yaml runtimeclass.node.k8s.io "youki" deleted deployment.apps "nginx-deployment" deleted -
Returning to the first command executed, the system calls youki issued are caught and logged.
$ just hack-bpftrace BPFTRACE_STRLEN=120 ./hack/debug.bt Attaching 13 probes... Tracing Youki syscalls... Hit Ctrl-C to end. TIME COMMAND PID EVENT CONTENT 207033348942 youki 13743 open errno=2, fd=-1, file=/opt/containerd/lib/glibc-hwcaps/x86-64-v3/libc.so.6 ... 207035462044 youki 13743 open errno=0, fd=3, file=/proc/self/exe 207035478523 youki 13743 write fd=4, ELF 207066996623 4 13743 open errno=2, fd=-1, file=/opt/containerd/lib/glibc-hwcaps/x86-64-v3/libc.so.6 ... 207070130175 4 13743 clone3 207070418829 youki:[1:INTER] 13747 write fd=4, {"timestamp":"2023-09-24T10:47:07.427846Z","level":"INFO","message":"cgroup manager V2 will be used","target":"libcgrou ... 207084948440 youki:[1:INTER] 13747 clone3 207085058811 youki:[1:INTER] 13747 write fd=4, {"timestamp":"2023-09-24T10:47:07.442502Z","level":"DEBUG","message":"sending init pid (Pid(1305))","target":"libcontai 207085343170 youki:[2:INIT] 13750 write fd=4, {"timestamp":"2023-09-24T10:47:07.442746Z","level":"DEBUG","message":"unshare or setns: LinuxNamespace { typ: Uts, path ... 207088256843 youki:[2:INIT] 13750 pivt_root new_root=/run/containerd/io.containerd.runtime.v2.task/k8s.io/0fea8cf5f8d1619a35ca67fd6fa73d8d7c8fc70ac2ed43ee2ac2f8610bb938f6/r, put_old=/run/containerd/io.containerd.runtime.v2.task/k8s.io/0fea8cf5f8d1619a35ca67fd6fa73d8d7c8fc70ac2ed43ee2ac2f8610bb938f6/r ... 207097207551 youki:[2:INIT] 13750 write fd=4, {"timestamp":"2023-09-24T10:47:07.454645Z","level":"DEBUG","message":"found executable in executor","executable":"\"/pa ... 207139391811 youki:[2:INIT] 13750 write fd=4, {"timestamp":"2023-09-24T10:47:07.496815Z","level":"DEBUG","message":"received: start container","target":"libcontainer 207139423243 youki:[2:INIT] 13750 write fd=4, {"timestamp":"2023-09-24T10:47:07.496868Z","level":"DEBUG","message":"executing workload with default handler","target"