Debugging Crossplane using VS Code

Recently I started to look at Crossplane, a CNCF sandbox project, which can extend the Kubernetes API to manage and compose infrastructure. One major feature provided by Crossplane is that it allows you to assemble infrastructure from multiple vendors and expose as higher level abstraction to consume without having to write any code. This is achieved by Crossplane along with its providers which extend Crossplane’s capabilities to support many different backends. The Crossplane community has a large number of providers available for you to choose. You can also create your own if there is no existing one that meets your specific need.

In order to better understand how Crossplane works, I was looking for an efficient way to read its source code. Instead of reading the code statically, people who get used to VS Code like me might consider live debugging since it can help you quickly understand what is going on at runtime by setting breakpoints, inspecting variables, and jumping across functions through the call stack.

After spending a few hours, finally I was able to debug Crossplane and its providers using VS Code. This post is going to share all necessary steps that I did to make it work including both local debugging and remote debugging.

Prerequisite

Just like many other existed cloud native projects in Kubernetes community, Crossplane is written in Go. All you need is VS Code with Go extension installed.

Besides that, in order to support remote debugging, you will need to install Delve, the Go debugger, separately on the remote machine. This is different from the one that you install locally with the VS Code Go extension by selecting Go: Install/Update Tools when you open the Command Palette. When debug remotely, instead of launching the local Delve instance, VS Code will send debugging commands to the Delve server you start on the remote machine. For more details on how to install standalone Delve, please refer to its installation doc.

Local Debug

Debugging Crossplane provider in VS Code locally is fairly easy. You just need to create a local debug configuration in launch.json as below:

{
    "name": "Launch file",
    "type": "go",
    "request": "launch",
    "mode": "debug",
    "program": "${file}",
    "args": [
        "--debug"
    ]
}

Many Crossplane providers accept input arguments. In above configuration, I added one argument --debug which is supported in many providers. It will print verbose logs to the console when debug the code which is very helpful for troubleshooting.

Since the main program entrance for many Crossplane providers starts from cmd/provider/main.go , to start debugging, you can expand the directory tree in VS Code and open main.go, then Select Run > Start Debugging from the main menu. After then, you will be able to debug the Crossplane code locally!

Remote Debug

In some cases, you may want to debug Crossplane remotely. There are many good reasons. For example, to me, I don’t want to put all workloads to my laptop so that can alleviate CPU and memory resource contention. With that, I only run VS Code on my laptop, and put all other workloads on a remote machine, usually a VM, where it runs Kubernetes cluster, usually kind as development environment, Crossplane runtime, and the providers.

Start Delve on remote machine

Before you start to debug provider on the remote machine, make sure you have already installed Delve on that machine, then start Delve as a headless server at root directory of your provider project by running below command:

dlv debug ./cmd/provider --headless --listen=:2345 --log --api-version=2 -- --debug

Here we specified the path to the package main source file: ./cmd/provider. We also specified the provider argument --debug which will be passed into provider code by Delve. Any arguments that you want to pass to the program you are debugging must also be passed to this Delve server.

Launch remote debugging on local machine

Once you have started the Delve server on remote machine, open VS Code on local machine and create a remote debug configuration in your launch.json as below:

{
    "name": "Launch remote",
    "type": "go",
    "request": "attach",
    "mode": "remote",
    "remotePath": "/path/to/provider/root/directory",
    "port": 2345,
    "host": “192.168.0.101”,
    "trace": "verbose"
}

Make sure to configure the port and host settings to point to your remote machine. If you run the headless Delve server and VS Code on the same machine, host should be 127.0.0.1.

It is also important to make sure the remotePath should point to the absolute path of the provider code being debugged on the remote machine. In this case, it is the root directory of your provider project. Otherwise, even you can connect to the remote Delve server, it will not be able to stop at the breakpoints you set locally.

Troubleshooting

It is recommended that you enable trace log as above by setting "trace": "verbose" in your launch configuration. With that, you will be able to see the actual calls made to Delve in the Debug Console pane to aid in your troubleshooting. Actually this helped me a lot when I experienced the issue of not being able to stop at the breakpoints that I set during the remote debugging. After I enabled the verbose logs, I learned that is because I did not configure the remotePath correctly when set the remote debug configuration.

Now that you have learned all the tips, happy debugging your Crossplane in VS Code!