Setting Up NetBSD Kernel Dev Environment

This is an overview of my setup:

  1. Linux Host With Qemu Target
  2. Tracing and Debugging using qemu’s built-in gdb server.
  3. pkgin for simpicity.(Sometimes have to use pkg_add to get stuff done)

Host Configuration

Make sure you have the latest version of qemu installed as we will be using x86-64 NetBSD guests.

We will be needing GDB that is configured with NetBSD x86_64 abi. So we need to compile it ourself.

1
2
3
4
5
wget http://ftp.gnu.org/gnu/gdb/gdb-xxxx.tar.xz
tar xvf gdb-xxxx.tar.xz
sudo mkdir -p /opt && cd gdb-xxxx
./configure --prefix=/opt --target=x86_64-netbsd
make -j8 && sudo make install

Building from NetBSD-current

First step get the files.

1
2
3
mkdir netbsd && cd netbsd
git clone https://github.com/NetBSD/src
cd src

Now for the time taking part, compiling the sources.

1
2
./build.sh -m amd64 -T ../tooldir -D ../destdir -R ../releasedir -O ../objdir -U -j6 release iso-image
# Now get some sleep it'll take some time.

Upon completion the directories will have the following files:

  • Cross-compilation toolchain in “tooldir”
  • Bootable image in “releasedir/images”

Now getting the guest up and running

We will be using virt-manager a front-end to qemu for configuring the vm, makes things easier without the hassle of setting up network bridges and all.

1
sudo apt install virt-manager

Once installed open Virtual Machine Manager

new vm
2
3
4
5
6

This should drop you a vm instance with the network capabilities too. Now go through the standard netbsd installation procedure installing all the necessary installation sets(depends on what you want to work with) and reboot.

Configuring ssh

1
2
cd /etc/ssh/
vi sshd_config

here you can configure the ssh port and other settings and make sure you have

1
PermitRootLogin yes

if you want to login as root. After making all the necessary changes,

1
service sshd restart

This restarts sshd and now you can ssh into your vm.

Installing pkgin

Once you are up and running better to install pkgin. It makes package management easier but sometimes it messes up in netbsd-current so if you have any problems with installing via pkgin, use pkg_add.

1
2
3
4
5
6
7
su -
export PKG_URL="http://cdn.netbsd.org/pub/pkgsrc/packages/NetBSD/amd64/9.0_2019Q4/All"
# change the above url accordingly
pkg_add "$PKG_URL/pkgin-0.9.4nb6.tgz"
#change the above url there will be a newer pkgin version out there.
echo $PKG_URL > /usr/pkg/etc/pkgin/repositories.conf
pkgin update

Now we can install all (well..most of em) the necessary utilities that makes life easier.

Compiling kernels

NetBSD runs the default GENERIC configuration. So we make a few changes to this and compile our own kernel.

1
2
cd ~/netbsd/src/sys/arch/amd64/conf
cp GENERIC QEMU

Use the text editor of your choice and fiddle with the configuration.

make sure you have

1
makeoptions     DEBUG="-g"      # compile full symbol table for CTF

Now all that is left is compiling.

1
2
./build.sh -m amd64 -T ../tooldir -D ../destdir -R ../releasedir -O ../objdir -U -u -j6 kernel=QEMU
# This should complete fairly quick

Now scp the files to vm and repace the old kernel

1
2
3
4
5
6
scp ~/netbsd/objdir/sys/arch/amd64/compile/QEMU/netbsd [email protected]<ip addr>:~
ssh [email protected]<ip addr>
# Virt-manager configures the network so just do and ifconfig get the ip address.
cp /netbsd /neetbsd.old
cp ~/netbsd /netbsd
# Reboot with the new kernel

Debugging with gdb

Start VM with qemu’s gdb stub forwarding tcp through port 1234.

1
2
3
qemu-system-x86_64 -drive file=vhd/netbsd-current.img,format=raw \
-m 1024 -enable-kvm \
-gdb tcp::1234

We already compiled netbsd kernel with complete symbol table.

1
2
cd ~/netbsd/objdir/sys/arch/amd64/compile/QEMU
/opt/bin/x86_64-netbsd-gdb ./netbsd.gdb

Now just simply enter

1
2
3
(gdb) target remote localhost:1234
Remote debugging using localhost:1234
0xffffffff8021d16e in x86_stihlt ()