BWFWTools - Perl scripts for manipulating Beyonwiz firmware

[prl]

I've just posted the collection of perl scripts that I've mentioned on this forum as BWFWToolson www.beyonwizsoftware.net.

Code: Select all

bw_rootfs        - extract or update the root filesystem romfs image in an
                   uncompressed Beyonwiz kernel.
getksyms         - extract the kernel module symbol table from an uncompressed
                   Beyonwiz kernel
gunzip_bflt      - convert gzip-compressed bFLT executable files into
                   uncompressed bFLT
make_kernel_bflt - convert a Beyonwiz kernel image into a bFLT executable
wrp_hdrs         - print the header information in Beyonwiz .wrp firmware
                   update files

BWFW             - Perl package of support routines for uncompressed Beyonwiz
                   kernel images.

doc\             - Documentation in plain text
html\            - Documentation in html (index in index.html)

Only extensively tested on Cygwin. Will probably work on Windows (will need a Perl installation), Linux, Mac OS X and other Unix variants.

bw_rootfs can easily create an unbootable system.

[tonymy01]

This sounds very handy "bw_rootfs" as my gzipping and hex cutting/pasting (and genromfs running) hand work the other day did not give expected results (i.e. it gave a non-booting system).
Regards

[prl]

This sounds very handy "bw_rootfs" as my gzipping and hex cutting/pasting (and genromfs running) hand work the other day did not give expected results (i.e. it gave a non-booting system).
Regards

Just a quick word of warning - repeating what's in the bw_rootfs documentation - I haven't actually constructed a .wrp with a modified root using it, and then booted from it. I've tested fairly carefully, but don't be surprised if it creates a BW killer .wrp

[tonymy01]

Hi Peter, the ROMFS extract (bw_rootfs) seems to not extract the whole ROMFS.
By just truncating the beginning gaff out of linux.bin and loopmounting that file, I see the mounted usr/local/bin directory contains:
root@tonysLinux:/mnt/beyonwiztoolchain/BWFWTools/romfsd/usr/local/bin# l
total 0
drwxr-xr-x 1 root root 32 1970-01-01 10:00 ./
drwxr-xr-x 1 root root 32 1970-01-01 10:00 ../
lrwxrwxrwx 1 root root 17 1970-01-01 10:00 auto_wnet -> /flash/bin/rompkg
lrwxrwxrwx 1 root root 17 1970-01-01 10:00 eepromtest -> /flash/bin/rompkg
lrwxrwxrwx 1 root root 17 1970-01-01 10:00 fastunzip -> /flash/bin/rompkg
lrwxrwxrwx 1 root root 17 1970-01-01 10:00 flash_eraseall -> /flash/bin/rompkg
lrwxrwxrwx 1 root root 17 1970-01-01 10:00 flash_update -> /flash/bin/rompkg
lrwxrwxrwx 1 root root 17 1970-01-01 10:00 flash_write -> /flash/bin/rompkg
lrwxrwxrwx 1 root root 17 1970-01-01 10:00 iwconfig -> /flash/bin/rompkg
lrwxrwxrwx 1 root root 17 1970-01-01 10:00 iwlist -> /flash/bin/rompkg
lrwxrwxrwx 1 root root 17 1970-01-01 10:00 iwpriv -> /flash/bin/rompkg
lrwxrwxrwx 1 root root 17 1970-01-01 10:00 mount.cifs -> /flash/bin/rompkg
lrwxrwxrwx 1 root root 17 1970-01-01 10:00 setgpio -> /flash/bin/rompkg
lrwxrwxrwx 1 root root 17 1970-01-01 10:00 setuartgpio -> /flash/bin/rompkg
lrwxrwxrwx 1 root root 17 1970-01-01 10:00 sil9030test -> /flash/bin/rompkg
lrwxrwxrwx 1 root root 17 1970-01-01 10:00 sys_up -> /flash/bin/rompkg
lrwxrwxrwx 1 root root 17 1970-01-01 10:00 testtest -> /flash/bin/rompkg

Where as with your PERL script extracted ROMFS, I see only:
root@tonysLinux:/mnt/beyonwiztoolchain/BWFWTools/romfsd/usr/local/bin# l
ls: cannot read symbolic link testtest: Input/output error
total 0
drwxr-xr-x 1 root root 32 1970-01-01 10:00 ./
drwxr-xr-x 1 root root 32 1970-01-01 10:00 ../
lrwxrwxrwx 1 root root 17 1970-01-01 10:00 auto_wnet -> /flash/bin/rompkg
lrwxrwxrwx 1 root root 17 1970-01-01 10:00 flash_eraseall -> /flash/bin/rompkg
lrwxrwxrwx 1 root root 17 1970-01-01 10:00 flash_update -> /flash/bin/rompkg
lrwxrwxrwx 1 root root 17 1970-01-01 10:00 flash_write -> /flash/bin/rompkg
lrwxrwxrwx 1 root root 17 1970-01-01 10:00 iwconfig -> /flash/bin/rompkg
lrwxrwxrwx 1 root root 17 1970-01-01 10:00 iwlist -> /flash/bin/rompkg
lrwxrwxrwx 1 root root 17 1970-01-01 10:00 iwpriv -> /flash/bin/rompkg
lrwxrwxrwx 1 root root 17 1970-01-01 10:00 testtest

So there is something wrong there.
Also just purging the beginning out of linux.bin, the file size is 18,135,408 bytes for .197 S1, where as your extraction generates a file 16,313,184 bytes. I can see there is still definitely parts of the filesystem in the linux.bin file that are not in the bw_rootfs generated file just by scanning past where your script stops in the linux.bin file.
I will still give it a go to see if it can update the image inside linux.bin better, but I am not holding my breath yet, as I didn't realise that your extraction didn't extract the files, but rather a singe file (the ROMFS image) which I have to loop mount, copy somewhere, modify, and then rely on genromfs to build something again that is going to be a legal ROMFS file to use your perl script to put back into linux.bin.

Regards

[tonymy01]

Ok, I can see the end of linux.bin isn't actually part of the filesystem, but the update part of the perl script works perfectly Peter, well done!
Hey f/w hackers, have you noticed the 4th image in the setup menu on the right (the "cogs") dissapears when not highlighted? Well, it happens for me, using wiz_unpack.exe from within windows to do this bit ? About to compile Erics unpack/pack for my linux environ now rather than using windows to see if this fixes this small quirk.
I thought it was because I browsed the image directory with Windows and it did something with one of the images, but I have done the win unpacking, tweaking, linux genromfs, win packing process twice now with the same behaviour.
Regards

[DaveR]

Either your 'Manage-unselect.png' or 'manage-unselect.png' file must be corrupt or missing. An early version of Eric's tools used to create a few corrupted png files.

I notice that (in 195 at least) there is both Manage-unselect.png and manage-unselect.png. So with Eric's tools in Windows this extracts them as 001x_Manage-unselect.png and 002x_manage-unselect.png. It seems to me that whatever you're using to repack the firmware is not correctly handling 2 files with different case spelling.

[prl]

Tony, from what you post, it looks like I'm dropping a block from the (end?) of the image. Thanks for the heads up. I'll have a look at it and see if I can find out what's breaking.

I'm glad the update side works properly!

[prl]

Hi Peter, the ROMFS extract (bw_rootfs) seems to not extract the whole ROMFS.
By just truncating the beginning gaff out of linux.bin and loopmounting that file, I see the mounted usr/local/bin directory contains:
root@tonysLinux:/mnt/beyonwiztoolchain/BWFWTools/romfsd/usr/local/bin# l
total 0
drwxr-xr-x 1 root root 32 1970-01-01 10:00 ./
drwxr-xr-x 1 root root 32 1970-01-01 10:00 ../
lrwxrwxrwx 1 root root 17 1970-01-01 10:00 auto_wnet -> /flash/bin/rompkg
lrwxrwxrwx 1 root root 17 1970-01-01 10:00 eepromtest -> /flash/bin/rompkg
lrwxrwxrwx 1 root root 17 1970-01-01 10:00 fastunzip -> /flash/bin/rompkg
lrwxrwxrwx 1 root root 17 1970-01-01 10:00 flash_eraseall -> /flash/bin/rompkg
lrwxrwxrwx 1 root root 17 1970-01-01 10:00 flash_update -> /flash/bin/rompkg
lrwxrwxrwx 1 root root 17 1970-01-01 10:00 flash_write -> /flash/bin/rompkg
lrwxrwxrwx 1 root root 17 1970-01-01 10:00 iwconfig -> /flash/bin/rompkg
lrwxrwxrwx 1 root root 17 1970-01-01 10:00 iwlist -> /flash/bin/rompkg
lrwxrwxrwx 1 root root 17 1970-01-01 10:00 iwpriv -> /flash/bin/rompkg
lrwxrwxrwx 1 root root 17 1970-01-01 10:00 mount.cifs -> /flash/bin/rompkg
lrwxrwxrwx 1 root root 17 1970-01-01 10:00 setgpio -> /flash/bin/rompkg
lrwxrwxrwx 1 root root 17 1970-01-01 10:00 setuartgpio -> /flash/bin/rompkg
lrwxrwxrwx 1 root root 17 1970-01-01 10:00 sil9030test -> /flash/bin/rompkg
lrwxrwxrwx 1 root root 17 1970-01-01 10:00 sys_up -> /flash/bin/rompkg
lrwxrwxrwx 1 root root 17 1970-01-01 10:00 testtest -> /flash/bin/rompkg

Where as with your PERL script extracted ROMFS, I see only:
root@tonysLinux:/mnt/beyonwiztoolchain/BWFWTools/romfsd/usr/local/bin# l
ls: cannot read symbolic link testtest: Input/output error
total 0
drwxr-xr-x 1 root root 32 1970-01-01 10:00 ./
drwxr-xr-x 1 root root 32 1970-01-01 10:00 ../
lrwxrwxrwx 1 root root 17 1970-01-01 10:00 auto_wnet -> /flash/bin/rompkg
lrwxrwxrwx 1 root root 17 1970-01-01 10:00 flash_eraseall -> /flash/bin/rompkg
lrwxrwxrwx 1 root root 17 1970-01-01 10:00 flash_update -> /flash/bin/rompkg
lrwxrwxrwx 1 root root 17 1970-01-01 10:00 flash_write -> /flash/bin/rompkg
lrwxrwxrwx 1 root root 17 1970-01-01 10:00 iwconfig -> /flash/bin/rompkg
lrwxrwxrwx 1 root root 17 1970-01-01 10:00 iwlist -> /flash/bin/rompkg
lrwxrwxrwx 1 root root 17 1970-01-01 10:00 iwpriv -> /flash/bin/rompkg
lrwxrwxrwx 1 root root 17 1970-01-01 10:00 testtest

So there is something wrong there.
Also just purging the beginning out of linux.bin, the file size is 18,135,408 bytes for .197 S1, where as your extraction generates a file 16,313,184 bytes. I can see there is still definitely parts of the filesystem in the linux.bin file that are not in the bw_rootfs generated file just by scanning past where your script stops in the linux.bin file.
I will still give it a go to see if it can update the image inside linux.bin better, but I am not holding my breath yet, as I didn't realise that your extraction didn't extract the files, but rather a singe file (the ROMFS image) which I have to loop mount, copy somewhere, modify, and then rely on genromfs to build something again that is going to be a legal ROMFS file to use your perl script to put back into linux.bin.

Regards

When I extract the root fs using bw_rootfs and wiz_unpack, I get:

Code: Select all

prl$ ls -l DPS1_Firmware_28Dec2007_ver_01.05.197.root1/usr/local/bin
total 120
lrwxr-xr-x  1 prl  prl  17 Mar  9 08:44 auto_wnet -> /flash/bin/rompkg
lrwxr-xr-x  1 prl  prl  17 Mar  9 08:44 eepromtest -> /flash/bin/rompkg
lrwxr-xr-x  1 prl  prl  17 Mar  9 08:44 fastunzip -> /flash/bin/rompkg
lrwxr-xr-x  1 prl  prl  17 Mar  9 08:44 flash_eraseall -> /flash/bin/rompkg
lrwxr-xr-x  1 prl  prl  17 Mar  9 08:44 flash_update -> /flash/bin/rompkg
lrwxr-xr-x  1 prl  prl  17 Mar  9 08:44 flash_write -> /flash/bin/rompkg
lrwxr-xr-x  1 prl  prl  17 Mar  9 08:44 iwconfig -> /flash/bin/rompkg
lrwxr-xr-x  1 prl  prl  17 Mar  9 08:44 iwlist -> /flash/bin/rompkg
lrwxr-xr-x  1 prl  prl  17 Mar  9 08:44 iwpriv -> /flash/bin/rompkg
lrwxr-xr-x  1 prl  prl  17 Mar  9 08:44 mount.cifs -> /flash/bin/rompkg
lrwxr-xr-x  1 prl  prl  17 Mar  9 08:44 setgpio -> /flash/bin/rompkg
lrwxr-xr-x  1 prl  prl  17 Mar  9 08:44 setuartgpio -> /flash/bin/rompkg
lrwxr-xr-x  1 prl  prl  17 Mar  9 08:44 sil9030test -> /flash/bin/rompkg
lrwxr-xr-x  1 prl  prl  17 Mar  9 08:44 sys_up -> /flash/bin/rompkg
lrwxr-xr-x  1 prl  prl  17 Mar  9 08:44 testtest -> /flash/bin/rompkg

The symlinks you are missing are in the last bit of the file system. Are you also missing the directories flash, proc, root and opt from the top level-directory when you unpack?

I also get a file size of 16,313,184 bytes for the file extracted from linux.bin (1.05.197) using bw_rootfs.

I did an ls -R of the directories extracted by bw_rootfs, and with my old tool that just strips the front part off linux.bin (18,135,408 bytes), and the two listings are identical.

I've just realised what the problem is! When you loopmount, the driver does all its I/O in whole (4096-byte?) blocks. The root romfs I extract isn't packed out to a full block boundary, it's just the bytes in the actual filesystem. I've used a modified (see below) BW-Background-Changer/WizTools/src/wiz_unpack to unpack, and that works just fine. Could you try that and see whether you get all the files (you may need to patch the distributed wiz_unpack)?

I think the best solution would be to add an option to pad to the 4k block boundary so that loopback mounting works. Any opinions on that, anyone?

If you want to modify and repack the root filesystem, then it needs to be unpacked. Unfortunately the wiz_unpack in the version of BW-Background-Changer I have doesn't extract symlinks. I've modified my copy to do that. Here are the context diffs for the mod.

Code: Select all

*** BW-Background-Changer/WizTools/src/wiz_unpack.c     Tue Dec 11 08:12:30 2007
--- BW-Background-Changer/WizTools/src/wiz_unpack.c.orig        Tue Nov 20 22:00:08 2007
***************
*** 183,195 ****
                        chmod(ni.name, mode);
                        break;
                case ROMFH_LNK:
!                       printf("symbolic link: %s\n", (char*)ni.file);
!                       if(symlink((char*)ni.file, ni.name) < 0) {
!                               fprintf(stderr, "can't link `%s' to `%s': %s\n",
!                                       ni.name, (char*)ni.file,
!                                       strerror(errno));
!                               return -1;
!                       }
                        break;
                case ROMFH_BLK:
                        printf("block device\n");
--- 183,189 ----
                        chmod(ni.name, mode);
                        break;
                case ROMFH_LNK:
!                       printf("symbolic link\n");
                        break;
                case ROMFH_BLK:
                        printf("block device\n");

Actually, I'm now wondering how BW-Backgroud-Changer works if wiz_unpack doesn't extract symlinks!

[prl]

I've just checked the genromfs code, and it always makes files rounded up to 1024 bytes, so the current behaviour of bw_rootfs is a bug, and I'll fix it to round the file up to the 1024-byte boundary on extraction. That will remove any need for an option.

Mac OS X can't mount romfs, so I'll have to post the mod without actually having tested it on a loopback mount. I will check that it does round up correctly!

[tonymy01]

Background changer doesn't expand the root/ROMFS at all, as the backgrounds are in the main wrp effectively (so not in the linux.bin.gz file is what I am trying to say).
The only way I can edit things in the root file system is to loopmount the romfs file, and cp -r the files to a folder I can edit, then genromfs, then run your perl update option.
This works fine thankfully, although it is a heck of a lot of steps (I have to genromfs twice actually, once to rebuild the root system file structure into a file your perl script will take, and once to rebuild the wizdvp & linux.bin.gz (and the few other bits and pieces like backgrounds etc) into a file in prep for the wiz_pack.
I was wrong about where your filesystem was truncated also, it appears the text strings after where you truncate the linux.bin.gz are strings that appear in the running f/w in the /proc directory.... the first one being /proc/version

I did all this to add some lines to rc.sysinit (I could have hex edited the linux.bin file to achieve the same, this is how I originally did the telnet patch, but it would mean purging some info out of it so I don't change its byte size), and to remove the busybox unmount and wrap it into a "killall wizremote; busybox unmount $1 $2 $3 $4" shell script.
This shell script worked, but it appears that Wizdvp has an unmount compiled into it, because my script wasn't called when I attempted a drive check Tis a shame.
So it appears the best solution for running some stuff off the HDD is to create a small partition there, that way the unmount and drive check routines etc won't break due to open files on the partition that wizdvp is trying to check.
Regards

[prl]

Hi, Tony. I knew that the background changer only modified the .wrp file ROMFS. What I'd forgotten was that this only works because the .wrp's ROMFS doesn't have any symlinks in it. The root ROMFS has loads of them.

Eric, could you release a version of WizTools that has my symlink patch added? That would let it be used for manipulating the root ROMFS as well as the .wrp ROMFS?

[tonymy01]

Either your 'Manage-unselect.png' or 'manage-unselect.png' file must be corrupt or missing. An early version of Eric's tools used to create a few corrupted png files.
I notice that (in 195 at least) there is both Manage-unselect.png and manage-unselect.png. So with Eric's tools in Windows this extracts them as 001x_Manage-unselect.png and 002x_manage-unselect.png. It seems to me that whatever you're using to repack the firmware is not correctly handling 2 files with different case spelling.

Ahh, ok, it is a case problem. I see all the other select/unselect files start with lower case, but this one image starts with uppercase. I wondered how that could be, and you are suggesting that both filenames exist, one upper, one lower, this may be why? It appears the unpack doesn't cope with this somehow? This is how I run unpack:
wiz_unpack -x untouched DPS1_Firmware_28Dec2007_ver_01.05.197.wrp.... edit: oops, but even though I compiled wiz_unpack to work on ubuntu, I was unpacking onto a samba share LOL.. maybe this is why!
So I don't use the "-i" option as I didn't want files in the fileystem to be modified with proceeding chars. It appears Eric has coded the unpack to work in winblows and given winblows doesn't destinguish the case differences, won't work (maybe). But I compiled it in ubuntu, and ran it there to see if this would fix the problem, and it was the same issue actually. Maybe I did need to run with the -i option, and in an operating system that will happily support Upper vs upper for filenames together in the same directory will not prepend any gaff onto the files. Perhaps his tool does a "if file exists, prepend gaff" check? I should look at the source shouldn't I!!
I will rename the file to lowercase "m" and this should solve the problem, but why the 2 files with upper/lower case leading char anyway, when the uppercase one isn't used?

edit:
Well look here, so why is the uppercase one the one that stays? It must see the file exists and not overwrite it:
root@tonysLinux:/mnt/beyonwiztoolchain/BWFWTools# wiz_unpack -x untouched2 DPS1_Firmware_28Dec2007_ver_01.05.197.wrp |grep unselect
photo-unselect.png: regular file
movie-unselect.png: regular file
Manage-unselect.png: regular file
DP-P1_unselect.png: regular file
music-unselect.png: regular file
manage-unselect.png: regular file
DP-S1_unselect.png: regular file
mis-unselect.png: regular file
network-unselect.png: regular file

Regards

[prl]

...
Well look here, so why is the uppercase one the one that stays? It must see the file exists and not overwrite it:
root@tonysLinux:/mnt/beyonwiztoolchain/BWFWTools# wiz_unpack -x untouched2 DPS1_Firmware_28Dec2007_ver_01.05.197.wrp |grep unselect
photo-unselect.png: regular file
movie-unselect.png: regular file
Manage-unselect.png: regular file
DP-P1_unselect.png: regular file
music-unselect.png: regular file
manage-unselect.png: regular file
DP-S1_unselect.png: regular file
mis-unselect.png: regular file
network-unselect.png: regular file

Regards

Only Manage-unselect.png appears in the extracted files when the .wrp is extracted into a case-insensitive-filenames HFS+ file system too. Bloody computer systems that won't do what they're told!

[DaveR]

but why the 2 files with upper/lower case leading char anyway, when the uppercase one isn't used?

Looks like someone in Korea made a typo when naming the file. And then somone copied a correctly named file into the build directory without realising that the new file would not overwrite the old one, or maybe they weren't aware that there were now 2 files.

Eric put a lot of effort into getting his tools to work correctly on a dumb file system. The win32 build now has the -i switch hard coded so Windows users can't do what Tony did. But from what I've seen this one png file is the only one to be aware of.

[DaveR]

So it appears the best solution for running some stuff off the HDD is to create a small partition there, that way the unmount and drive check routines etc won't break due to open files on the partition that wizdvp is trying to check.

Wouldn't it be better to run stuff off a USB stick? This way the user could do a 'Format HDD' or replace their HDD etc without needing to repartition and reload stuff. If we did it right they could even remove the USB stick before booting to disable whatever is on the USB stick.

This way we could make downloadable USB stick images that average users can easily load onto their USB sticks to try some of this stuff. It just seems so much cleaner and simpler if 99.9% of the stuff is on a USB stick.

[prl]

...
The only way I can edit things in the root file system is to loopmount the romfs file, and cp -r the files to a folder I can edit, then genromfs, then run your perl update option.
This works fine thankfully, although it is a heck of a lot of steps (I have to genromfs twice actually, once to rebuild the root system file structure into a file your perl script will take, and once to rebuild the wizdvp & linux.bin.gz (and the few other bits and pieces like backgrounds etc) into a file in prep for the wiz_pack.
...

I'm in the process of writing perl scripts to unpack and repack the .wrp file to/from the flash file system and, optionally, the root file system. They will use wiz_unpack and genromfs-win32 so they will work on systems that don't support mounting ROMFS file systems (Windows and OS X, for example). I probably won't get to finish them until the Easter weekend.

I'll put an option in both to allow '-i' to be passed through from the script to wiz_unpack and genromfs-win32, but of course using that option will have repercussions on any scripts or hand modifications made to the file systems by patchers.

[tonymy01]

I don't know how you will get this to work as Windows doesn't support soft links? Or perhaps inside a cygwin environ they are supported, not really sure?
Regards

[prl]

I don't know how you will get this to work as Windows doesn't support soft links? Or perhaps inside a cygwin environ they are supported, not really sure?
Regards

But Windows does have soft links. It just calls them Shortcuts. They can do more that Unix soft links, but they can do everything that soft links do. Cygwin generates a Shortcut to emulate a Unix soft link.

Eric has added my patch to wiz_unpack.c that creates the symlinks; I originally wrote and tested that code under Cygwin. I think that Eric's code needs cygwin1.dll to run on bare Windows (without a full Cygwin install), so that should take care of the symlink creation in the same way.

[cwiggles]

Actually Windows does support soft/hard links via the linkd command. This MS article explains it for those interested. It is one of the things MS doesn' seem to advertise much for some reason.

http://support.microsoft.com/kb/205524

I think it's part of the windows resource kit so you may have to download the resource kit to get it. I'm pretty sure it's available for XP as well as NT200/2003/2008. The only real constraints is that it is only suported on NTFS and not FAT, though apparently the taget can exist on at FAT based partion.

Craig

[peteru]

Be careful with junction/reparse points and other forms of "links" on Windows systems. I've heard a horror story just earlier this week about how badly Windows Explorer handles them and that their use can lead to irrecoverable loss of data.

[prl]

I've not (yet) had problems with shortcuts created by Cygwin, but then I don't use Windows Explorer much in my Cygwin tree.

[DaveR]

I've played with Windows version of symbolic links in Windows Server 2003. I hated them. IIRC they screwed up defragging, virus scanning and searching etc because the processes kept going around in circles, or maybe it just did everything twice. I'd perform some action on one drive, or directory, and it'd follow the links to a whole other drive or directory and perform the action there as well. But for our purposes none of this should matter...

[prl]

A new version of BWFWTools is almost ready to go. It will include a "Make install" that may make using it a bit easier to use (that bit still to write).

It will also include

• unpack_wrp and pack_wrp, scripts that assemble together the best bits of BWFWTools and Erics wizfwtools into single scripts to unpack a .wrp into the flash (and optionally) the root file system, and to pack a flash (and optionally root) file system into a .wrp file. All in one command each , though pack_wrp has too many options.
• Made the algorithm for finding the root ROMFS in the kernel more robust, and in particular, made it possible to unpack the root ROMFS in 1.05.211 beta.
• Fixed a bug that was making bw_rootfs incorrectly claim that the padding area after the root ROMFS in the kernel wasn't zeroed wneh it in fact was (fix below).

There's a bug in wiz_genromfs -i that creates mangled directory names for what should be the '.' and '..' entries. I've sent Eric a patch. In the meantime don't use wiz_genromfs on case-insensitive file systems, and avoid the -i option.

Fix for "not zeroed" bug in bw_rootfs.
In check_lastfs_block change:

Code: Select all

    $kpos = $kpos + roundup($size, BLOCKSZ);

to

Code: Select all

    $kpos = $kpos + roundup($size, BLOCKSZ) - BLOCKSZ;

That's what I get for making "trivial" changes without testing.

[prl]

New version now available on http://www.beyonwizsoftware.net/.

Version 0.1.2

Code: Select all

unpack_wrp:  new tool
pack_wrp:    new tool
README.txt:  added
Makefile:    added
BWFW:        renamed to Beyonwiz::Kernel
bw_rootfs:   now works for firmware 1.05.211.
             root ROMFS location code made more robust.
             bug fixed in check that root ROMFS spare space is all zeros.
make_doc.sh: more fiddling with pod2html options to try to make links
             work properly
Beyonwiz::Kernel:
             Fixed context info for _end at 0x90092104
             Now gets address of blkdev memory arean instead of directly
             fetching the root ROMFS location - more stable.

unpack_wrp options firmware_file flash_directory [root_directory]

Extracts the files in firmware_file (Beyonwiz .wrp file) into
the directory flash_directory as they would appear in the Beyonwiz's
/flash directory.
If the root_directory argument is also given, the Beyonwix root
file system is extracted from the kernel image in flash_directory
into that directory.

pack_wrp options firmware_file flash_directory [root_directory]

Packs the files in flash_directory, and optionally, in root_directory
into firmware_file (Beyonwiz .wrp file).

If the root_directory file is given, linux.bin.gz in
flash_directory is ungzipped and the ROMFS root filesystem constructed
from root_directory is patched into the kernel, and the kernel then recompressed.

[DaveR]

Nice. Did your previous versions also extract and reconstruct the ROMFS?

[prl]

Nice. Did your previous versions also extract and reconstruct the ROMFS?

Thanks.

bw_rootfs extracts the root ROMFS out of an uncompressed Beyonwiz kernel, and can patch a replacement root ROMFS into the uncompressed kernel, but just as a ROMFS file, it doesn't expand/generate the ROMFS to/from a directory or compress/uncompress the kernel. As Tony previously noted, patching the root file system was a cumbersome multi-step process. unpack_wrp and pack_wrp are just wrapper scripts that make it simpler to do that process with already existing tools, some from Eric's wizfwtools.

So

Code: Select all

unpack_wrp fw.wrp flash.dir root.dir

is equivalent to

Code: Select all

wiz_unpack -x flash.dir fw.wrp
gunzip -c flash.dir/linux.bin.gz > flash.dir/linux.bin
bw_rootfs flash.dir/linux.bin flash.dir/root.romfs
wiz_unpack -x root.dir flash.dir/root.romfs
rm flash.dir/linux.bin flash.dir/root.romfs

... then do whatever changes you want in root.dir and/or flash.dir...
Then (assuming that fw.wrp was a DP-S1 firmware file)

Code: Select all

pack_wrp new_fw.wrp flash.dir root.dir

is equivalent to

Code: Select all

wiz_genromfs -V mambo -d root.dir -f flash.dir/root.romfs
gunzip flash.dir/linux.bin.gz > flash.dir/linux.bin
bw_rootfs -u flash.dir/linux.bin flash.dir/root.romfs
gzip -c flash.dir/linux.bin flash.dir/linux.bin.gz
rm flash.dir/linux.bin flash.dir/root.romfs
wiz_genromfs -V mambo -d flash.dir -f /tmp/flashnnn
wiz_pack -t s -V xxx.yyy.zzz_wiz_pack -i /tmp/flashnnn -o new_fw.wrp
rm /tmp/flashnnn

The values for the "-t s" and "-V xxx.yyy.zzz_wiz_pack" are generated from the name of the file in flash.dir/version. There's an option to wrp_pack to oeerride the "__wiz_pack" suffix of the firmware version number. The "-v" version number is what's displayed in the verification popup when you download the firmware to the Beyonwiz.

If you don't want to make changes to the root directory, just leave root.dir off when you run unpack_wrp and pack_wrp.

The *pack_wrp scripts pretty much run the programs as printed (varying the options according to options passed to them), except that some of the g*zips are actually implemented using a Perl module. All the wiz_* programs are part of Eric's wizfwtools. The dependence on Eric's tools is in the documentation.

It's all about reuse

[DaveR]

thanks for the indepth explaination, Peter. Saves me reading the script to see what it's doing.

[rwhitby]

I have imported the BWFWTools source code (all released versions) into an openwiz.org SVN repository at

http://svn.openwiz.org/svnroot/bwfwtools/trunk

The versions are tagged in:

http://svn.openwiz.org/svnroot/bwfwtools/tags

There is also a wiki and bug reporting facility at:

http://trac.openwiz.org/bwfwtools

As soon as I confirm the name that Peter wants put in his ssl client certificate, I'll send him that certificate so that he has write access to this repository.

If anyone would like write access to this svn repository, just let me know.

-- Rod

[prl]

A couple of questions about BWFWTools:

1. Has anyone taken the plunge and tried booting a .wrp generated from wrp_unpack/wrp_pack? I assume that the silence so far on this means that no-one has tried it and failed, but has anyone actually succeeded?
2. Has anyone used BWFWTools on Windows (without Cygwin)? Any suggestions how it might be improved for that environment, like BAT file wrappers to run the scripts?
3. I'm thinking of writing another layer on top of wrp_unpack/wrp_pack. The idea would be to pass a script the name of an input.wrp, an output .wrp and a list of Perl scripts to run on the unpacked firmware. By mixing and matching the scripts that are specified on the command line, you can make a customised hacked download. Does that sound useful? I'd probably start by producing a script to do the telnet hack.

[tonymy01]

I haven't tried it, but I can't see why it wouldn't work. I managed to edit a file in a romfs mounted version (and copied to a writable location) of the linux.rom.gz file and used genromfs to build it all back up using about a 3 step process or so and had a booting Wiz at the end of all that, about a week before you brought out this release. I have the telnet mod now simply in rc.local which means it survives firmware upgrades with stock firmware (thanks Eric for this idea) and thus haven't needed to rip apart a f/w image for a little while.
Why don't you try it????? Can't you get access to a PC at all (even a dodgy $50 laptop or something?) to use the tftp windows uploader to fix a potential failing to boot wiz?

[prl]

bump...

[prl]

...
Why don't you try it????? ...

Having an actual life as well as a forum life?

Can't you get access to a PC at all (even a dodgy $50 laptop or something?) to use the tftp windows uploader to fix a potential failing to boot wiz?

I intend giving it a go this weekend. I'm not quite the Windowsophobe I may seem at times

[IanSav]

Hi Guys,

I must admit that I use my DP-S1 too much to want to risk playing with hacked firmware.

It also concerns me that using modified firmware won't really help test and verify the firmware that Beyonwiz gives some of us to test.

Further, with Beyonwiz starting to work on more rapid releases it is going to be a hard task to keep building, installing and following the Beyonwiz releases. Generally I would start looking at hacked firmware when Beyonwiz gets their code more feature complete and release updates less frequently.

Regards,
Ian.

[tonymy01]

You don't have to hack the f/w anymore (well, just for simple telnet anyway) once you get an rc.local in to enable telnet. That stays in the config area which is kept during upgrades and called by the stock f/w.
Regards

[prl]

There's a new version, 0.1.3, of BWFWTools on beyonwizsoftware.net.

Added a printout of spare space in the firmware flash to wrp_hdrs.

Added new tool, bwhack, that allows some hacks (including telnet) to be enabled and disabled remotely. A companion file, rc.local, to install on the Beyonwiz in /tmp/config, that checks the flag files set by bwhack to actually implement the enabling/disabling.

bwhack and its rc.local are intended to be easy to extend to other hacks. They currently enable/disable front panel dimming, telnet access, HTTP access to the whole BW filesystem, and wizremote.

Tested on DP-S1, should work unmodified on DP-P1, guaranteed not to work on DP-H1.

The next step is a tool to make a telnet-enabled .wrp file, so anyone can make their own Telnet Patch. It was supposed to get done this weekend, but I released a buggy version of getWizPnP

[Kezza]

The next step is a tool to make a telnet-enabled .wrp file, so anyone can make their own Telnet Patch. It was supposed to get done this weekend, but I released a buggy version of getWizPnP

so hows this coming along?

Kezza

[prl]

The next step is a tool to make a telnet-enabled .wrp file, so anyone can make their own Telnet Patch. It was supposed to get done this weekend, but I released a buggy version of getWizPnP

so hows this coming along?

Kezza

It's been stalled for a few weeks. Real Life? intervened. The current state of play is:

• I have a new version of Eric's Wiz Firmware Tools checked in to the OpenWiz repository, and a ready-to-go Windows version of the tools awaiting final testing before I get Eric to post them on http://www.beyonwizsoftware.net/.
• I have checked a version of BWFWTools into the OpenWiz repository that has a new tool, "bw_patcher.pl" with accompanying patcher modules that let it install the Telnet patch alone, my patch to work with bwhack.pl to allow patches to be switched on and off, and my version of Tony's USB hack. The modules allow a single command line to prepare a patched firmware .wrp
• I also have a new patcher module for "bw_patcher.pl" that updates the background, like BWBackground changer, only command-line based and portable.
• Updated the documentation to include better installation advice, and put in consistent warnings about the hazards of hacking BW firmware.

My ToDo is:

• Rewrite the background patcher to be a module that calls a generic "put this file into the firmware there" module; that also means there will be a generic patcher module to put any file anywhere in the firmware.
• Test all the new bits more throughly on Windows.
• Some more documentation tidying-up
• Make it all available on the OpenWiz Repository and on BeyonwizSoftware.

Depending on how colour scheme switching works in the new beta, background changer hacks may have to be rethought anyway.

[IanSav]

Hi Peter,

Depending on how colour scheme switching works in the new beta, background changer hacks may have to be rethought anyway.

We should know more tonight, if things go to plan.

Regards,
Ian.

[prl]

A new version of BWFWTools is availible on the OpenWiz Subversion repository (http://trac.openwiz.org/bwfwtools) and on BeyonwizSoftware.net (http://www.beyonwizsoftware.net/software-b28/bwfwtools/). The ZIP version on BeyonwizSoftware.net contains Windows and Cygwin binaries of all the .exe files used by BWFWTools so it is ready for use on those syatems. On other systems, a copy of Eric Fry's Wiz Firmware Tools needs to be built and installed.

This version of BWFWTools will not work with earlier ZIP releases of Wiz Firmware Tools. In particular, if you use it with WizFwTools_20071204.zip, it may create patched firmware that is known to render a Beyonwiz unstartable! Until the head posting in the Wiz Firmware Tools topic is updated by the post owner, you must use the version at the head of the Subversion repository or in the Download link on my post in the Wiz Firmware Tools topic. The likns above will take you to the correct versions.

A set of Perl scripts for manipulating Beyonwiz firmware in various hopefully useful ways.

Version: 0.2
Author: prl

Contents of the zip:

• Makefile - Unix make file for install/uninstall/build documentation
• README.txt - some useful information about installing BWFWTools on Unix-like systems
• bw_rootfs - extract or update the root filesystem romfs image in an uncompressed Beyonwiz kernel
• bwhack - remotely enable and disable Beyonwiz "hacks"
• getksyms - extract the kernel module symbol table from an uncompressed Beyonwiz kernel
• gunzip_bflt - convert gzip-compressed bFLT executable files into uncompressed bFLT
• make_kernel_bflt - convert a Beyonwiz kernel image into a bFLT executable
• pack_wrp - pack a Beyonwiz firmware update file
• unpack_wrp - unpack a Beyonwiz firmware update file
• bw_patcher - all-in-one tool to automatically apply Beyonwiz firmware patches
• wrp_hdrs - print the header information in Beyonwiz .wrp firmware update files
• svcdat - print the contents of Beyonwiz C<svc.dat> (service scan configuration) files
• Beyonwiz::Kernel - Perl package of support routines for uncompressed Beyonwiz kernel images
• Beyonwiz::Hack - contains patcher modules to use with bw_patcher
• Beyonwiz::Hack::BackgroundChanger - change the background image used in the File Player and Setup screens
• Beyonwiz::Hack::BwhackSupport - Support for hacks that can be turned on and off remotely using bwhack
• Beyonwiz::Hack::PutFile - Put a single file into an existing directory in the firmware
• Beyonwiz::Hack::RemFile - Remove a file from the firmware
• Beyonwiz::Hack::Telnet - Enable the telnet daemon in the firmware (allows remote logins to the Beyonwiz using telnet)
• Beyonwiz::Hack::USBHackSupport - Allows hacks to be run from a USB stick or memory card
• make_doc.sh - Unix shell script to generate HTML and text documentation from embedded POD markup
• doc\ - Automatically generated documentation in plain text
• html\ - Automatically generated documentation in html (index in index.html)
• etc\ - Beyonwiz firmware patches that can be used in conjunction with Beyonwiz::Hack::USBHackSupport and Beyonwiz::Hack::BwhackSupport
• wiz_pack.exe, wiz_genromfs.exe & wiz_unpack.exe - compiled Cygwin and Windows binaries of some of Eric Fry's Wiz Firmware Tools. Used by some of the BWFWTools.
• gzip.exe - GNU gzip executable for Windows. Used by some of the BWFWTools.
• cygwin1.dll - Cygwin Linux emulation DLL. Used by wiz_pack.exe, wiz_genromfs.exe & wiz_unpack.exe on Windows. Not needed in other environments

Only extensively tested on Cygwin & Mac OS X. Some testing on Windows. Will probably work on Linux and other Unix variants.

WARNING: bw_rootfs, pack_wrp and bw_patcher can all easily create an unbootable system.

Release notes
bw_patcher: new tool
svcdat: new tool
getksyms: fails gracefully on firmware versions 01.05.269 beta and later, which don't have a kernel symbol table for module loading.
wrp_headers: Print out system ids in the same way that they appear on the Beyonwiz.
bw_rootfs: prints more information about free space in the firmware, updating code a bit more robust

In several tools, the way that sub-tools are called has been changed to avoid possible problems with shell quoting, and a bug in Cygwin Perl 5.10.0 IO::Uncompress::Gunzip::gunzip() has bee worked around.

Bugs:
pack_wrp cannot delete one of its temporary files on Windows. A warning is pinted, and the file can be deleted manually.

WARNING

Using user extensions or hacks may make your Beyonwiz unable to operate correctly, or even start. Some modifications are known to interfere with the correct functioning of the Beyonwiz.

If your Beyonwiz cannot start after you load modified firmware, you may need to use the procedures in the NOTICE - How to recover from FW update failure procedure on the Beyonwiz forum. It's not known whether that procedure will fix all failures due to user modifications or "hacks".

If you run modified firmware on your Beyonwiz, and have problems with its operation, try to reproduce any problems you do have on a Beyonwiz running unmodified firmware, or at least mention the modifications you use when reporting the problem to Beyonwiz support
or on the Beyonwiz Forum. Beyonwiz support may not be able to assist if you are running anything other than unmodified firmware from Beyonwiz. Forum contributers may be able to be more flexible, but they will need to know what modifications you have made.

[craigh]

This new version worked without a hitch on Xp.

Installed ActivePearl then BWFWTools.

Pointed it at my existing FW and in a minute had a new version with telnet enabled.

Fantastic.

[prl]

This new version worked without a hitch on Xp.

Installed ActivePearl then BWFWTools.

Pointed it at my existing FW and in a minute had a new version with telnet enabled.

Fantastic.

Thanks!

[jim16]

Hi prl,

Is there anything special needed to patch the FW for the DPP2? I can successfully run this tool against the firmware for the DPP1 (DPP1_Firmware_07Aug2008_ver_01.05.261.wrp) but everytime I try it against the DPP2 (DPP2_Firmware_07Aug2008_ver_01.05.261.wrp) fw it reports:

..\PatchedFw.wrp is too big to fit in BW flash memory: 8093696 > 8060928
- firmware .wrp file not created

Code: Select all

bw_patcher.pl -c 9 ..\DPP2_Firmware_07Aug2008_ver_01.05.261.wrp ..\PatchedFw.wrp Beyonwiz::Hack::Telnet

Is -c 9 the highest compression available? I tried 10 but it produced errors.

jim16

[tonymy01]

The P2 has double the flash size of the P1, so it should be fine. I guess Peter just hard coded a P1/S1/H1 flash size into the warning/check code.
Regards

[prl]

Hi prl,

Is there anything special needed to patch the FW for the DPP2? I can successfully run this tool against the firmware for the DPP1 (DPP1_Firmware_07Aug2008_ver_01.05.261.wrp) but everytime I try it against the DPP2 (DPP2_Firmware_07Aug2008_ver_01.05.261.wrp) fw it reports:

..\PatchedFw.wrp is too big to fit in BW flash memory: 8093696 > 8060928
- firmware .wrp file not created

Code: Select all

bw_patcher.pl -c 9 ..\DPP2_Firmware_07Aug2008_ver_01.05.261.wrp ..\PatchedFw.wrp Beyonwiz::Hack::Telnet

Is -c 9 the highest compression available? I tried 10 but it produced errors.

jim16

9 is the maximum compression available in gzip, and the BWFWTools have used that level of compression as the default for a while, since it appears that Beyonwiz has used gzip -9 to compress the firmware for a while, too (earlier Beyonwiz firmware was compressed with the gzip default -6).

The problem is that while the three earlier Beyonwiz models (DP-S1, DP-P1 and DP-H1) only have 8060928 (0x7b0000) bytes available for the firmware, the DP-P2 has a bigger flash (16MB?), and apparently the firmware is bigger - I'm not sure why.

If you have a look at the output of bw_patcher, you'll see that the original firmware is bigger than the 8060928 limit for other models anyway:

Code: Select all

romfs image size: 8092640

I'll need to have a look in the P2 kernel to find out the sizes of its flash partitions and provide an option to check against the P2's flash size instead of the other models' size.

The 1.05.280 firmware for the P2 is also bigger than 8060928 bytes, so I can't offer that as a workaround.

I'll try to get a new version that includes a fix to this problem (as well as being able to patch the beta releases post-1.05.280 out on the weekend.

You're probably the first person to try using BWFWTools on a P2 (at least you're the first to tell me it doesn't work!). I strongly recommend that you try the firmware recovery procedure before you try using the patched firmware in case the patch fails. I will, of course, check that the patcher modules put their changes in the right places in the P2 firmware, but I can't actually test on a P2.

[jim16]

Hi prl,

Thanks for looking into this and developing this easy to use tool. I have been meaning to patch the FW on my P2 for quite sometime now but only just managed to find the time and motivation ( Ice subscription running out soon ).

jim16

[prl]

...
The problem is that while the three earlier Beyonwiz models (DP-S1, DP-P1 and DP-H1) only have 8060928 (0x7b0000) bytes available for the firmware, the DP-P2 has a bigger flash (16MB?), and apparently the firmware is bigger - I'm not sure why.
...

The DP-P2 has 16449536 (0xfb0000) bytes available for firmware. It has the same amount of space, 327680 bytes (5 x 64kiB blocks), reserved for other purposes (config & boot loader) as the older Beyonwiz models.

bw_patcher.pl (or actually pack_wrp.pl, which handles the creation of the new .wrp file from the modified unpacked firmware) discovers (or you've told it) what machine the firmware is for, so it's relatively easy (and should be invisible to the user) to set the appropriate flash size to test against.

[prl]

...
bw_patcher.pl (or actually pack_wrp.pl, which handles the creation of the new .wrp file from the modified unpacked firmware) discovers (or you've told it) what machine the firmware is for, so it's relatively easy (and should be invisible to the user) to set the appropriate flash size to test against.

pack_wrp.pl (and hence bw_patcher.pl) now sets the maximum size of the firmware module appropriately for the flash size for different models, and it can correctly re-pack the DP-P2 01.05.261 firmware and test its size against the correct flash memory size. This should normally be completely automatic, there should be no need to specify any command-line options.

Still needs some more testing, especially checking that the "hooks" to run the hacks get inserted in the right places for the DP-P2.

[DaveR]

the DP-P2 has a bigger flash (16MB?), and apparently the firmware is bigger - I'm not sure why.

A/V Input (LiDiC) module/code?

[prl]

the DP-P2 has a bigger flash (16MB?), and apparently the firmware is bigger - I'm not sure why.

A/V Input (LiDiC) module/code?

That seems most likely, though LiDiC support is in the S1 and P1 firmware, too.