Project Title & GitHub Link

Hacking Unitree Cameras

Technologies

Unitree, strace, GDB

Project

This post (still in progress) details efforts to completely eliminate the use of the UnitreeCameraSDK to stream from the onboard USB cameras on the Unitree Go1 robot. The goal was to completely replace the Jetson Nanos on the robot with fresh copies that don’t contain any Unitree software. This was complicated by the fact that the onboard cameras are locked until the UnitreeCameraSDK gets run, and this SDK is closed source.

I was able to pinpoint the exact syscall at which the cameras are enabled through the UnitreeCameraSDK source code using strace and gdb. It is an ioctl call of the following format:

ioctl(5, UVCIOC_CTRL_QUERY, &query)

5 is a file descriptor referring to /dev/video0, the streaming device about to be unlocked
UVCIOC_CTRL_QUERY queries and sets various control parameters of UVC compliant USB video devices
query is a struct of type uvc_xu_control_query, defined in /usr/include/linux/uvcvideo.h. I found the fields of this struct to be:
```
query.unit = 3
query.selector = 5
query.query = 1
query.size = 64
```

This syscall happens after a long stream of UVIOC_CTRL_QUERY that sets camera parameters. These are likely to be vendor specific and it is unclear specifically which of the proceeding queries are necessary for unlocking.

I wanted to create a simple C script that would perform the appropriate syscalls to unlock the cameras without the use of the UnitreeCameraSDK, but was spending too much time on this and needed to focus on other aspects of my final project using this robot. I currently have a simple script that relies on their camera SDK, and will unlock the cameras by instantiating a camera object and promptly segfaulting the program (more info below). After this, you can stream from the cameras using native Linux tools like ffmpeg or ROS packages like usb_cam (work in progress on these steps for ROS 2 here).

I believe that completely eliminating the UnitreeCameraSDK from the Unitree Go1 allows for easier use of their onboard cameras and promotes open source development! If you have questions, ideas, or additional work that would contribute to this effort, please feel free to reach out to me.

Details to replicate

First, connect to a Jetson Nano on the Go1. Make sure that you can see USB cameras via lsusb and video devices with ls /dev/video*. My standard way of testing the cameras was through the linux ffmpeg command on /dev/video0 (the same will hold for /dev/video1).

$ ffmpeg -i /dev/video0 output0.mp4

If the cameras are “locked”, this command will hang, and when you CTRL-C you will see the following error message with no data being encoded. Any other attempts to read from the cameras will similarly fail with no data being encoded.

could not find codec parameters for stream 0 (Video: mjpeg, none(bt470bg/unknown/unknown), 1856s800): unspecified pixel format

If the cameras are “unlocked”, ffmpeg will start streaming data.

Autostart

The first thing was determining exactly what was enabling the cameras via the Unitree/autostart folder. If you disable this entire folder on a Jetson Nano by renaming it something else, completely power the dog off and on again, the cameras will be locked. The same thing will happen if you put a completely fresh Jetson Nano into the main board of the dog.

If you allow autostarting, then kill the camera processes manually and run the same command, the cameras are unlocked. Note that if you don’t kill the camera processes, ffmpeg will fail because the video devices are occupied.

Next, I renamed only Unitree/autostart/camerarosnode and performed another hard poweroff. The cameras were locked, indicating that whatever was enabling the cameras was instantiated there. This pushed me to investigate the UnitreeCameraSDK outside of the context of ROS, since one of the first thing that node does is create a UnitreeCam object.

UnitreeCameraSDK

All of this further investigation was done with the Unitree/autostart folder completely disabled. Upon boot, the cameras are locked. If you run any of the example scripts from UnitreeCameraSDK (say example_getRawFrame), and exit them cleanly (by pressing ESC), the cameras will stay locked. If you exit abruptly, with CTRL-C or by killing the process through a terminal, the cameras will be unlocked.

This indicated to me that something in the UnitreeCam object constructor was setting appropriate camera parameters and something in the destructor was un-setting them. When the destructor gets called, it prints out:

“Unitree camera resource has been released”

which occurs after the camera parameters have been un-set.

I created two incredibly simple C++ functions to test this hypothesis. The first (below) results in device /dev/video0 being locked as the destructor is able to get called.

int main{
	UnitreeCamera cam(0);
	return 1;
}

The following unlocks /dev/video0 by segfaulting!

int main{
	UnitreeCamera cam(0);
	abort();
	return 1;
}

Equivalent functions can be used to unlock /dev/video1 by instantiating UnitreeCamera cam(1).

With an incredibly minimal program for testing, it was time to do a deep dive into what was happening in the instantiation.

strace and GDB

strace is a way of tracing system calls of a running process to the Linux kernel. I obtained straces of the locking and unlocking code above. To replicate this yourself, you can clone my fork of the UnitreeCameraSDK. Once you have built the project, navigate to UnitreeCameraSDK/bins and you can run:

$ strace -f -v -s 1000 -o strace_locking.txt ./lock_camera0
$ strace -f -v -s 1000 -o strace_unlocking.txt ./unlock_camera0

Make sure the camera in question is locked before running these, so that you can actually track the syscalls that unlock it! If it is unlocked, you can lock it using the ./lock_cameraX executable.

I first wanted to see what was being called in the locking code that wasn’t being called in the unlock code. Using vscode’s compare files was really useful for me here. Unfortunately, it was difficult to parse by hand, with the trace from the unlock code at around 4100 syscalls and the locking code at around 4700 syscalls.

I next went into gdb to step through line by line to try to figure out which syscall actually enabled the cameras. From an alternative test ran with my advisor Matthew Elwin, we discovered that attempts to read from a locked camera using pure Linux syscalls in C hung on an ioctl(VIDIOC_DQBUF)(source code for this test here). We figured ioctl’s were a good starting point to try to narrow down the source of the instantiation as they manipulate device parameters, and we saw a good number of them in the straces.

To pinpoint the ioctl at which the camera got unlocked, I performed the following procedure. First, I wanted to know how many ioctls gdb encountered while executing this program.

$ ./lock_camera0          # ensure camera is locked before starting
$ gdb unlock_camera0      # start gdb
$ b ioctl                 # set a breakpoint on ioctl, should return id 1
$ ignore 1 1000000        # ignore next [very large number] crossings of breakpoint 
$ r                       # program should complete since the ignore limit was so high
$ info b 1                # should tell you how many ioctl's it hit

Next I performed a binary search to figure out the exact ioctl the cameras get enabled on. I could then coordinate this with the list of ioctl’s I obtained from strace to see the arguments more clearly.

$ ./lock_camera0          
$ gdb unlock_camera0      
$ b ioctl                 
$ ignore 1 ${total_ioctls/2}        
$ r                                     # should stop on middle ioctl
$ info b 1                              
$ q                                     # quit gdb
$ ffmpeg -i /dev/video0 out0.mp4        # test if cameras unlocked

Depending on whether the camera is locked or unlocked, you can repeat this process again but change the number of times the breakpoint is ignored to eventually zero in on the exact ioctl. In my case, the cameras were enabled on the 177th out of 363 ioctl’s in the unlock code. This was the same number of ioctl’s in my strace, so I was able to pinpoint it in context. It happens at the end of a very long string of UVCIOC_CTRL_QUERYs:

...
ioctl(5, UVCIOC_CTRL_QUERY, 0x7fe3250a08) = 0
ioctl(5, UVCIOC_CTRL_QUERY, 0x7fe3250a08) = 0
ioctl(5, UVCIOC_CTRL_QUERY, 0x7fe3250a08) = 0
ioctl(5, UVCIOC_CTRL_QUERY, 0x7fe3250a08) = 0
ioctl(5, UVCIOC_CTRL_QUERY, 0x7fe3250a08) = 0
ioctl(5, UVCIOC_CTRL_QUERY, 0x7fe3250a08) = 0
ioctl(5, UVCIOC_CTRL_QUERY, 0x7fe3250a08) = 0
ioctl(5, UVCIOC_CTRL_QUERY, 0x7fe3250a08) = 0
ioctl(5, UVCIOC_CTRL_QUERY, 0x7fe3250a08) = 0
nanosleep({tv_sec=0, tv_nsec=1000000}, NULL) = 0
ioctl(5, UVCIOC_CTRL_QUERY, 0x7fe3250990) = 0
nanosleep({tv_sec=0, tv_nsec=1000000}, NULL) = 0
ioctl(5, UVCIOC_CTRL_QUERY, 0x7fe3250990) = 0                     # Camera enabled here!!!
close(5)                          = 0
...

Now that I had found the ioctl that enabled the cameras, I needed to figure out what data was in the address at the third argument. This involved some more reading into how ioctl and UVIOC_CTRL_QUERY work. The third argument is a pointer to uvc_xu_control_query, defined in /usr/include/linux/uvcvideo.h. On arm64, the registers x0, x1, and x2 are supposed to contain the first, second, and third arguments of the syscall. Thus, the contents of the struct can be printed out with

print (struct uvc_xu_control_query) *$x2

From this, I see that the fields are:

query.unit = 3
query.selector = 5    
query.query = 1       # corresponds to UVC_SET_CUR in /usr/include/linux/usb/video.h
query.size = 64 

I tried adding a query with this information to the C video streaming code, but this did not enable the cameras. It is likely that there are a series of UVCIOC_CTRL_QUERYs that are necessary to set the appropriate camera parameters. Since there are so many of these queries, it is hard for me to go by them through hand to understand what is actually important.

Repos/Files

Acknowledgements

Thanks to Matthew Elwin for all the help and Linux expertise he provided for this project.

Katie Hughes

Resume LinkedIn Github