Using 7020 software on 7000 - the miniroot project

[noggie]

Hi all,

The image that isn't....

Here's my solution to how to run the new 7020-software on the 7000, using e.g. a USB stick to compensate for the lack of internal flash in the 7000. Using the attached software, I'm currently running the pre-release "image" for the 7020 on my 7000 from flash and USB. I haven't run it long enough to comment how stable it is, but I would be surprised and impressed if I don't find some (more) bugs as I keep experimenting. BTW, the included software also allows me to run Hydra Zeus in parallell with the 7020 image, I just have to change a line in a config file to switch between them. Not quite as cool as FlashWizard, but then FlashWizard doesn't allow me to run software for a machine that isn't even released yet :-) My third partition on the USB is currently free, while the fourth is used for swap.

Please read attached README's for more info. The small attachment contains just the README/INSTALL files plus a copy of the important shell scripts from miniroot. It's suitable for those who just want to get an overview of what this is all about. The large attachment contains everything (including the READMEs).

I do *not* intend to update this package as the pre-release 7020 software gets closer to actual release. Do it yourself if that's what you want, or wait for the real image makers. This is just a proof-of-concept, my answer to the up-coming problem of how 7000 owners can take advantage of the software that will be developed for the new series of machines, and how we can avoid (at least for a little while) being left behind by all developers flocking to the bright, shiny, exciting world of the new software development environment. The new cdk really *is* a big step forward.

Run and install it if you want, but please read all the warnings first.

Last minute warning: After I flashed miniroot, I can't seem to get dreamUp to work anymore. I have absolutely no idea what could cause this, it just doesn't make sense. It could be related to the image, or it could be something entirely different. You've been warned...

[andriko]

heh - cool !!!!

[noggie]

Originally Posted by andriko

heh - cool !!!!

Yeah, I think so too. Not as much what I did, but what the new dev. kit allows me to do. I've just spent a few hours looking at it so far, but already I've discovered that the ncurses build that I distributed may be flawed(*), I've recompiled it, re-installed it using the new package system, and then compiled and installed vim with just a few easily solved problems. Finally a real "vi" instead of the brain-dead version that's part of busybox!(**) :-) All in a few hours time and starting with near-zero prior knowledge of openembedded. The future indeed looks bright for independant developers!

-- noggie

(*) I compiled the cdk (just to see if I could) in mid-November. Since then, the glibc has been upgraded. Seems that something in libncurses are dependant on the exact version of glibc used. Since glibc was upgraded and recompiled in my late December build, while ncurses wasn't, "vim" wouldn't link properly. Can't see this problem with any of the installed packages, and it's simple enough to recompile ncurses and install it using the "ipkg" system.

(**) To make the comparison between busybox-vi and vim fair, it must be mentioned that vim alone is 50% larger than the whole of busybox. And that's without the needed support libraries and files...

[PsiDOC]

Noggie,
Brilliant work. I had often wondered about this myself. We all know the 7020 has a seperate bootloader so I am assuming you use the original flash as the bootloader then switch to the USB drive for the software? Or am I way wide of the mark?
As an afterthought my research on the DM 7000 shows it may be able to take a 64 MB bootable flash rom as an onboard replacement. Obviously soldering skills are required kids! I have yet to try it, however it's a work in progress and still at the drawing board stage.

If it works, maybe the 7020 bootloader can be added to the flash bootblock and 7020 images run on the 7000 with an upgraded flash rom?

Just a thought.
regards,
Psi

[noggie]

Originally Posted by PsiDOC

Noggie,
Brilliant work. I had often wondered about this myself. We all know the 7020 has a seperate bootloader so I am assuming you use the original flash as the bootloader then switch to the USB drive for the software? Or am I way wide of the mark?

Thanks for the kind words, although I feel I don't deserve them.

If I understand your question correctly, you're spot on. I haven't touched the original bootloader, I'm just introducing one more step in the boot process (the miniroot) so that external storage can be used for the image. It's very much like using "initrd", only that flash is used instead of RAM for the initial root file system.

Originally Posted by PsiDOC

As an afterthought my research on the DM 7000 shows it may be able to take a 64 MB bootable flash rom as an onboard replacement. Obviously soldering skills are required kids! I have yet to try it, however it's a work in progress and still at the drawing board stage.

If it works, maybe the 7020 bootloader can be added to the flash bootblock and 7020 images run on the 7000 with an upgraded flash rom?

Just a thought.
regards,
Psi

Interesting thought. You would need to re-compile the kernel (the partitioning of the flash is hard-coded), but as long as the config file from Dream is available, that shouldn't be very difficult. From what little I've looked at this, the 7020 flash consists of three partitions:
- a small secondary bootloader (a binary exists for this)
- a boot partition (kernel and boot pictures)
- the root partition
Both the boot and the root partions are jffs2.
So your 64MB flash could have a fourth partition (the 7020 flash seems to be 32MB), jfffs2-formatted, and remain compatible with 7020 images.

Please keep us updated if do find out that upgrading internal flash is within reach of anybody with some soldering skills.

Did you look at internal RAM also, BTW? That's what's becoming the serious limiting factor for more complex software. Already, I could easily enough replace the busybox with the full-featured version of each utility, as fas as disk space is concerned, but the dreambox would start swapping like mad if I tried it. Same goes for more complex windowing software etc.

[PsiDOC]

Noggie:
I understand what you mean about the RAM. Problem is the onboard RAM on the Dream has quite literally no room for expansion. There's no room to drop larger chips in the DreamBox and adding to the existing is a no-go as ther's no spare space for expansion.
Sorry to be the harbinger of doom on this point, As much as I'd be happy to say yes it can expand the RAM it's can't due to the hardware.

The upgrade to the flash rom should be fairly easy from a hardware point. Chip out and (programmed) chip in. Obviously to run the 7020 firmware this would require the 7020 boot block in the flash and the software in the upper sections, as the 7000 firmware is now, Unlike the 7020 which has them seperate (IE: Boot block Bios and seperate flash for the firmware) Any help on combining the two on the one chip is appreciated. I don't do software too well, but give me hardware and I'm a whizz
Regards,
Psi

[Tom0000]

About ram...
If I am not wrong, prog/data 32MB of ram is used smaller address space. The driver's 32MB part is using bigger address space. If someone will replace buid-in chip with bigger one, the soft of dreambox will not be possible to address memory space outside of driver's part of ram.
Only moving driver's part adressing space is a way to increase prog/data ram part.

[noggie]

I had this idea that I just couldn't put away until I tried it... I don't need it, but it I thought it could be cool to see if I could pull it off.

In priciple, by using a two-stage booting (using a initrd/miniroot as initial root partition, then switch to the real root partion) you should be able to boot with a root partition from CIFS. It took me a few attempts to set up (some extra boots to NFS to fix things), but I did make it in the end.

I'm running the miniroot in flash, so the standard bootloader is doing just the normal thing, getting a kernel from cramfs, and booting with sqashfs (the miniroot) as root.

Now miniroot takes over, uses some of the hooks that I've made available to extend the bootprocess from the jffs2 partition in flash, and basically does the following:
1) Loads the cifs.ko kernel module
2) Sets up the network (I'm using just a static IP address)
3) Mounts a Windows share using cifs.
4) Mounts a file on that share as an ext3 file system using the loopback feature.
5) Proceeds to the trickery of setting this last filesystem up as root.
6) Continues the boot of the 7020 software which I've loaded to that file. The 7020 software is slightly more hacked than it used to be, in that I just turned off network installation because it otherwise would interfere with the CIFS mount.

So now I can (on my standard 7000) run software for a machine that's not for sale yet (the 7020) running off a file on a windows machine. Except for the times that I needed to run off NFS to bail me out of the mistakes I made in setting it up, I wouldn't have needed a NFS server and Linux machine at all. All the setup could have been done from a PC and the miniroot. It would just be false modesty if I tried to claim that I don't find this way cool :-)

If anybody's interested in the details, please get in touch and I'll write down a few words. Don't expect a step-by-step recipe, though. It is (as I found out) easy to do a mistake where booting off NFS is the only alternative to re-flashing a standard image to get control over the machine again, so if you need very detailed instructions, you shouldn't be doing it.

There's one more detail to this story... My Windows machine isn't really a machine. It's just a virtual machine running off VMware on my Linux box... Now try to trace in your mind what that "read" system call in Enigma is actually going through before the data stored on my IDE disk is finally returned to the application... Makes you quite dizzy, eh? :-) :-)

[Tom0000]

I am reading and reading all what you have written ... I am not specialist of linux and kernel things ... but I want to understand one.

When we will boot system , it will boot from firmware in internal flash. Some kernel modules will be read and run, and on the end miniroot will be run. After replaceing "root" what will be do ? :
- firmware will continue to start using next modules
or
- firmware will unload used modules and will load all new modules from new "kernel root" changed during run miniroot
?

The idea of using CIFS is interesting. Write more, if you will find time.

[noggie]

If you forget for a minute about any special trickery like miniroot and NFS, and so on, and consider just the boot process of a standard image, the dreambox boots something like this:

1) The firmware starts and loads the kernel from flash (from the cramfs part) and also displays an initial picture on the TV.
2) The kernel starts running. The first part of the kernel image is really a loader program which unpacks the kernel (it is stored in compressed form in the flash), moves stuff around a bit, relocates some addresses etc. Then the kernel-loader displays a line of kernel options, allows the user (on the serial console) to change the kernel option, and jumps to the kernel proper if there is no response on the serial console.
3) The kernel does a minimum of initialization, and starts /sbin/init. In many ways, this is the end of the booting part. Files and programs on the root file system is now in control over which programs should be run.
4) A standard /sbin/init is controlled by /etc/inittab, and in our case, this says that /etc/init.d/rcS should be started. And so on and so on.

It can be quite illuminating to turn on console output and just read the messages produced during boot.

Connect a serial cable between the PC and the dreambox, and start "hyperterminal" on the PC. Hyperterminal should be set up with COM1, 115200 baud, no parity, 8 data, XON/XOFF handshake. It's also a good idea increase the number of lines of saved text in File->Properties->Settings->Backscroll buffer lines.

No restart your dreambox. You should see a line in hyperterminal saying "Dreambox DM7000". Now you have a second or so to press "RETURN". If your PC is too far away from your dreambox, you need some help, one person restarts the dreambox, the other sits tapping "RETURN" on the PC every second or so. By pressing "RETURN" at this point , the boot process is interrupted in point 1) of my list above. You'll get a menu of various things you can do, and you're in fact "talking" to the bootloader in the firmware. If you press "0", the boot process continues. By tapping the "space" key every second or so, you will get a new interruption in the boot process (in point 2) in my list above). You're now "talking" to the loader at the start of the linux kernel. You should see a line saying

Linux/PPC load: console=null root=/dev/mtdblock5 rootfstype=squashfs ro

and maybe an emtpy space or three at the end, which comes from your tapping the "space" key. Just backspace however much you need, leave one space at the end, and add "console=ttyS0,115200n8" to this line, (you don't need to remove the other "console=null" at the start of the line, the last console statement takes precedense in this case).

Press "RETURN", and read the boot messages as they scroll by. A large number of saved lines will make it possible to scroll back up and read all the stuff that scrolled by too fast for anyone to read.

Also, after the box is booted, telnetting in to it and running

dmesg | more

gives you most of the boot messages that was produced by the kernel during initialization.