pvrusb2 driver utilities

$Id$

Mike Isely <isely at pobox dot com>

The home location for this page is here.

This page describes the usage of the utilities included with the pvrusb2 driver distribution.

Overview

The utilities can be found in the util subdirectory. They include the following:

decode_log.cpp - This is a decompiler, which is used to interpret the USB traffic flowing between the pvrusb2 device and the driver.
fwextract.pl - This is a Perl program which can extract firmware data out of Hauppauge's Windows driver files.
old-extract-firmware.pl - This is the old firmware extractor created by Björn Danielsson. Use the newer fwextract.pl program instead; this is included here for historical reasons and also as a fallback in case problems are encountered with the new extractor.
usbreplayprep.pl - This is a Perl script which can postprocess USB sniffer output into a compact form that the decompiler can parse. This also originally came from Björn however it has been modified to also preserve data coming from the device (older version only preserved data going *to* the device).

Sniffing and decompiling USB traffic

There is a free USB sniffer that can be run under Windows for capturing all traffic between a USB device and its driver. The home page for it is here. Using this tool we can learn about how the Windows driver operates the device. If you want to explore, download that tool and follow the related instructions.

To help with understanding the log data from the sniffer, there are two tools provided with this driver:

The first tool is usbreplayprep.pl, which will preprocess the USB sniffer data into a more compact form. This tool was originally written by Björn but I have since modified it to provide more data. The version provided here saves both directions of traffic; the previous one only saved data going in the driver-to-device direction.

The second tool is decode_log, which is compiled from the similarly named C++ source file. This program interprets the output from usbreplayprep.pl and produces a concise listing of all the commands and data moving to / from the device and the driver. (This tool can also capture and save the encoder firmware if seen as part of the log.)

The decompiler is written in C++ and must be built first. Run make in the util directory to build this tool. Note that this program has no other dependencies beyond needing C++ so it should be a straight-forward build.

To capture and decompile a session with the device when run with the Windows driver, do the following:

Boot windows, start the sniffer and enable it, and plug in the device.
Run your TV app for a little while. NOTE: The sniffer log data can get ENORMOUS so you may not want to run it for long. Also the capture process does slow things down so the video playback in the TV app might get choppy, but that's OK.
Copy the log data to your Linux system or reboot to Linux
Run usbreplayprep.pl < data.log >data.txt where data.log is the name of the log output from the sniffer.
Run decode_log < data.txt > data.src and the decompiled output will be in data.src.

The decompiler also has an option for saving off the encoder firmware captured, which is useful when one has to figure out the firmware extraction details for a newer Hauppauge driver snapshot. There's more about that later on...

Firmware Automatic Extraction

The main pvrusb2 web page gives a recipe for quick extraction. The more general process is basically this:

Create an empty subdirectory somewhere (e.g. "win_driver") and unpack or copy the Windows driver files into it.
Execute the fwextract.pl with no arguments, specifying the driver directory you created with the --driverDir=name option. If you named that directory win_driver then you can omit the option.

Note that when the term "Windows driver files" is used above, it is refering to a directory full of files that contains the drivers from Hauppauge. That unfortunately is easier defined than found because Hauppauge seems to like packaging these drivers in different ways. If you're working from a CDROM distribution, then we're refering to the root directory of the CDROM. If you are working from a download package, there might be several such packages but only one has the files you want. For example, the packages pvrusb2_23_31351.exe and pvrusb2_27_23178.zip each contain a driver set, but pvrusb2_26_23055.exe does not - instead in that case the actual drivers are in pvrusb2_inf.zip. Why did they do this? I have no idea; it just confuses. Best advice here is make a guess and if fwextract.pl doesn't spot anything then try a related package.

What this all really boils down to is just a matter of expanding all the Windows driver files into a single place and then letting fwextract.pl chew on it for a while to locate the firmware. The extract script doesn't care about file names or history; it works purely by recognizing file MD5 sums and acting on what it finds.

Assuming that fwextract.pl recognizes the driver files, it will automatically locate and extract the firmware. This program uses MD5 sums to verify that the correct file(s) are being extracted and to verify that the resulting data is correct. It is reasonably smart and can't be easily fooled by wrong files / versions. No more firmware file guesswork (yay).

Firmware Manual Extraction

It is possible that fwextract.pl might not recognize your driver package no matter what you throw at it; you'll see messages of the form "failed to locate data for writing filename" (where filename is the firmware file it tried to generate). This can happen if Hauppauge releases new drivers and we haven't updated fwextract.pl yet. If this is indeed the case, then things are somewhat more difficult - we need to manually identify the firmware data and teach the script where to find it. (Note that the same fwextract.pl handles many driver versions so we stay backwards compatible all along.) Basically what has to be done if this happens is something like this:

Find some alternate means to get the firmware data.
Run "fwextract.pl --search".

The second step runs a process to scan the original Windows driver files and find where the firmware (previously fetched here in step 1) is hiding. Once found, the fwextract.pl will write a Perl expression to the terminal which contains information needed by the extractor for locating the firmware in the future. Send that snippet to me and I'll place it into the master copy of fwextract.pl so that other users can benefit from your hacking effort.

The first step here is obviously the big one. There are two approaches to it. One approach is that someone good at reverse-engineering starts hacking away with a hex editor to find the embedded data - which is what people have been doing all along anyway. Another approach is to capture the firmware from the hardware itself. There are two pieces of data to capture: the 8051's firmware and the encoder's firmware:

To manually capture the encoder's firmware, you can use the USB sniffer to grab it "in flight" from within Windows and then issue a command to the decompiler to write the data to a file. To do this, run decode_log with "--firmware filename" on the command line, setting filename to be "pvrusb2.f2". The decompiler will then write that file with encoder firmware data captured from the USB sniffer output (and filtered first through usbreplayprep.pl). Here's the exact procedure:
Boot up Windows and start the USB sniffer (see the previous section about USB sniffing to learn about where to find the sniffer). Plug in the pvrusb2 device. You don't need to run anything but wait for the sniffer to collect the log data from when the driver initializes the device.

Copy the sniffer's log data to some place where you can get at it from Linux.

Boot Linux.

Run this sequence (where "logfile.txt" is the file you got from the sniffer):
usbreplayprep.pl < logfile.txt | decode_log --firmware pvrusb2.f2 > /dev/null
Capturing of the 8051 firmware is entirely different. The USB sniffer doesn't see this traffic right now so we have to take a different strategy. I've added a feature to the pvrusb2 driver that makes possible the direct retrieval of the firmware data from the device. Here's the procedure:
Boot up Windows, power cycle the device, and then plug in the pvrusb2 device.

Reboot to Linux LEAVING THE PVRUSB2 DEVICE POWERED UP.

Execute this sequence (substitute your device's serial number for "XXXX"):
echo "cpufw fetch" >/sys/class/pvrusb2/sn-XXXX/debugcmd
cat /dev/video0 >/tmp/pvrusb2.f1
echo "cpufw done" >/sys/class/pvrusb2/sn-XXXX/debugcmd
It is imperative that you do not power cycle the device when switching from Windows to Linux. This allows the 8051 firmware to be retained, and will cause the pvrusb2 Linux driver to not attempt to reload the firmware. Thus when you run the command sequence to pull the firmware back, you'll get what had been loaded by Windows.

Your device will reset itself after the "cpufw done" debug command is sent to the driver. The file /tmp/pvrusb2.f1 is an image of the 8051's program memory. Note that this isn't the exact bit-for-bit copy of the firmware file you need; apparently the 8051 program modifies itself slightly. However it's close enough that the fwextract.pl program's --search feature can still locate the pristine firmware data using a fuzzy matching algorithm (which it will fall back to if an exact search doesn't find it).

After you've gotten pvrusb2.f1 and pvrusb2.f2 retrieved, put copies in the current directory, (if you haven't already done so) unpack the Hauppauage Windows driver package again into a empty (e.g. "win_driver") subdirectory, then run fwextract.pl --search. (If you installed from a cdrom, then mount the cdrom under Linux and use the --driverDir option to point at the mount point.) After a minute or so it will print the magic chunk of data that can then be put back inside fwextract.pl so it will now automatically understand how to extract this firmware data in the future. Send that magic chunk to me (along with a description of the driver package) and I'll ensure that the next release of fwextract.pl includes it.