[pvrusb2] Completely new fwextract.pl coming...

[Mike Isely]

All:

Just thought I'd drop a note that I'm preparing a rewritten 
fwextract.pl.  New features include:

* Native recognition of "naked" firmware files.  There will no longer be 
any need to train it in order to locate firmware image files "hiding in 
plain sight" within a collection of files.  The net effect is that 
fwextract.pl will find those files and pull them up, named appropriately 
for their purpose.  This means for example that now you can simply pull 
all the files from Steve Toth's page, blindly run fwextract.pl on it, 
and you'll get back a properly named set of firmware files for use with 
the pvrusb2 driver.

* Configuration data is now line-oriented text.  Previously fwextract.pl 
held firmware configuration data as a Perl data structure.  Now it's a 
series of text records.

* Configuration data is simply appended to the end of fwextract.pl.  
Previously after "training" fwextract.pl one had to edit the source file 
and paste in the new records into its internal structure.  No more.  Now 
all one has to do is this:

	echo >>fwextract.pl

and then just mouse-paste the lines and hit Ctrl-D (ends the echo 
command).  That's it!

* Merging of configuration data.  What this means is that if the 
training process finds stuff that it already knows about, it won't emit 
out any new records for that data.  Instead it will only emit just 
enough data to represent the "new" information that it has learned.

* Completely generic - the program is no longer hardcoded for the 
pvrusb2 driver.  All knowledge of the binary firmware, including file 
names, text descriptions, and locations is represented now as block of 
configuration data.  This, if there is interest, makes this program 
usable for anyone's extraction needs.

* A new --inventory option - run this and the program will dump out a 
concise summary of its configuration database.  (Useful for a developer 
to see what it knows about.)

* A new --exportDatabase option - this takes a file name argument.  
When run it will write that file with the configuration database.  Run 
this after --search and the new database will also include the new 
information learned from the training process.  If the file has ".db" 
for a suffix, the data written will be a Perl structure.  Otherwise it 
will be simple text configuration records - you can use that data to 
edit fwextract.pl and replace the internal configuration data appended 
to the end of the program.

* A new --importDatabase option - this takes a file name argument and 
uses that as its configuration database, substituting for whatever might 
have already been in the program.  Data format is the same as what is 
written by --exportDatabase.  This means you could also do the full 
training process like this:

	fwextract.pl --search --exportDatabase=modified.txt
	fwextract.pl --importDatabase=modified.txt

That writes the full database, with new records, to modified.txt, then 
you can use that database to extract the final firmware.

* Self-training - I have an idea in mind that should make it possible to 
train fwextract.pl on new driver packages withOUT the need for the 
firmware to be manually extracted first.  This trick will only work if 
the firmware data hasn't changed (which is frequently the case).  If new 
firmware data has been embedded, a manual extraction is still needed to 
provide the search samples for the training process.  This trick will 
work by using the internal database basically as the search set.

Will all of the above, the basic operation of the program has not 
changed.  All of the old behaviors still work as before.  If you run it 
with no options, it will do as it has always done - scan all files for 
known firmware and extract it.  If you run it with --search then it will 
perform a search and print out what it finds (but the data will be in 
the new, far more compact form).

I'm still doing a bunch of tweaks and fixes.  All but the last have been 
implemented and already work.  Hopefully tomorrow I can make this 
available.  I'd appreciate it if people here could then play with it and 
let me know of any problems.  The net effect of all this is that 
fwextract.pl should be even smarter now and for the common cases it 
should be even easier.

  -Mike




-- 

Mike Isely
isely @ isely (dot) net
PGP: 03 54 43 4D 75 E5 CC 92 71 16 01 E2 B5 F5 C1 E8