Setting Up NetBSD Kernel Dev Environment

20 April, 2020

I used T_PAGEFLT's blog post as a reference for setting my NetBSD kernel development environment since his website is down I'm putting down the steps here so it would be helpful for starters.

This is an overview of my setup:

Linux Host With Qemu Target
Tracing and Debugging using qemu's built-in gdb server.
Use a cronjob with rsync to keep my files updated b/w host and guest.
pkgin for simpicity.(Sometimes have to use pkg_add to get stuff done)

Host Configuration

Make sure you have the latest version of qemu installed as we will be using x86-64 NetBSD guests.

We will be needing GDB that is configured with NetBSD x86_64 abi. So we need to compile it ourself.

wget http://ftp.gnu.org/gnu/gdb/gdb-xxxx.tar.xz
tar xvf gdb-xxxx.tar.xz
sudo mkdir -p /opt && cd gdb-xxxx
./configure --prefix=/opt --target=x86_64-netbsd
make -j4 && sudo make install

Building from NetBSD-current

First step get the files.

mkdir netbsd && cd netbsd
git clone https://github.com/NetBSD/src
cd src

Now for the time taking part, compiling the sources.

./build.sh -m amd64 -T ../tooldir -D ../destdir -R ../releasedir -O ../objdir -U -j6 release iso-image
# Now get some sleep it'll take some time.

Upon completion the directories will have the following files:

Cross-compilation toolchain in "tooldir"
Bootable image in "releasedir/images"

Now getting the guest up and running

qemu-img create ~/vhd/netbsd-current.img 10G
qemu-system-x86_64 -smp 4 -drive file=vhd/netbsd-current.img,format=raw \
                     -cdrom ~/Code/netbsd-current/releasedir/images/NetBSD-8.99.12-amd64.iso\
		     -m 2048 -enable-kvm \
		     -net user,hostfwd=tcp::5022-:22 -n

No go through the standard installation without unnecessary clutter like x11,games etc. Configure all the necessary things such as ssh, users, sheel preference. After the completion of installation

qemu-system-x86_64 -smp 4 -drive file=vhd/netbsd-current.img,format=raw \
                     -m 2048 -enable-kvm \
		     -net user,hostfwd=tcp::5022-:22 -net nic

This should drop you a vm instance.

Installing pkgin

Once you are up and running better to install pkgin. It makes package management easier.

su -
export PKG_URL="http://cdn.netbsd.org/pub/pkgsrc/packages/NetBSD/amd64/9.0_2019Q4/All"
# change the above url accordingly
pkg_add "$PKG_URL/pkgin-0.9.4nb6.tgz"
echo $PKG_URL > /usr/pkg/etc/pkgin/repositories.conf
pkgin update

Now we can install all the necessary utilities that makes life easier.

Compiling kernels

NetBSD runs the default GENERIC configuration. So we make a few changes to this and compile our own kernel.

cd ~/netbsd/src/sys/arch/amd64/conf
cp GENERIC QEMU

Use the text editor of your choice and fiddle with the configuration.

make sure you have

makeoptions     DEBUG="-g"      # compile full symbol table for CTF

Now all that is left is compiling.

./build.sh -m amd64 -T ../tooldir -D ../destdir -R ../releasedir -O ../objdir -U -u -j6 kernel=QEMU
# This should complete fairly quick

Now scp the files to vm and repace the old kernel

scp -P 5022 ~/netbsd/objdir/sys/arch/amd64/compile/QEMU/netbsd root@localhost:~
ssh -p 5022 root@localhost
# We forwared VM's port 22 --> Host 5022 for ssh acceess
cp /netbsd /neetbsd.old
cp ~/netbsd /netbsd
# Reboot with the new kernel

Debugging with gdb

Start VM with qemu's gdb stub forwarding tcp through port 1234.

qemu-system-x86_64 -drive file=vhd/netbsd-current.img,format=raw \
                     -m 1024 -enable-kvm \
		    -gdb tcp::1234

We already compiled netbsd kernel with complete symbol table.

cd ~/netbsd/objdir/sys/arch/amd64/compile/QEMU
/opt/bin/x86_64-netbsd-gdb ./netbsd.gdb

Now just simply enter

(gdb) target remote localhost:1234
Remote debugging using localhost:1234
0xffffffff8021d16e in x86_stihlt ()