About that virtual factory reset...

... it did work out in the end, after a few problems. The DVD images worked great all the way to the subsequent reboot. Then all one got was a black screen with a non-blinking low cursor in the upper-left corner of the screen. Thanks a lot - I guess the factory reset never bothered to install any MBR...

Oh well, let's boot up the virtual machine again, but this time with a Linux live CD iso image attached (Linux Mint 16 "Petra"), so that we can - from the trustworthy Linux prompt - install the mbr package ('sudo apt-get install mbr') and use it to install an MBR on the harddrive of the virtual machine (/dev/sda in my case, checked in the output of dmesg): 'sudo install-mbr-i n -p D -t 0 /dev/sda' (check the man-page of install-mbr to identify the details of those standard options, they're basically no interaction, use first bootable partition, and use a timeout of zero). (We could have used other Linux means as well, for example the mbr.bin image provided by the syslinux package.) Now let's try another reboot.

Well, it was a so called "fall forward". Instead of the solid, low-cursor it now stood "MBR" in the upper-left corner and then we got to an error screen, complaining about winload.exe being 
missing or corrupt ("Status 0xc000000e", "Windows failed to start. A recent hardware or software change might be the cause.", "File: \Windows\System32\winload.exe", "Info: The selected entry could not be loaded because the application is missing or corrupt."). Bugger... Out of ideas, I turned to Google and found this eminent guide. By luck, I already had a Windows XP virtual guest system I could attach the new machine's virtual harddrive to - otherwise, I would probably not been able to get any further. Now, I could, from XP, follow the guide, i.e., in my case, the new harddrive was mapped as E:, so I could run E:\Windows\System32\bcdedit /store F:\Boot\BCD /enum to check if I suffered from the same situation as he had. I sure did. with three unknowns. I changed them all to "boot" according to his recipe (E:\Windows\System32\bcdedit /store F:\Boot\BCD /set {X} Y boot, where (X, Y) were (bootmgr, device), (default, device), and (default, osdevice), respectively.

Upon next boot, the new machine started-up and begun configuring up itself, completing the factory reset. Cool! 

I Stand Humbly Corrected...

So, I've been dabbing with trying to boot a set of "XYZ" recovery DVDs in a VirtualBox guest, to see if I can get a virtual factory reset running.

The first problem I ran into is that it seems that my USB-hub cannot support both the external USB-harddrive the virtual host resides on and the USB DVD reader at the same time - and, of course, I have too little space left on my internal drive and too few USB ports to have them directly attached (well, I could direct attach one with the inconvenience of unplugging the external mouse and keyboard, but we can't have that, can we?).

Thus, I needed to convert the 4.2 GB DVD to an ISO-image - but both Copy Disk in Brasero and old, honsest dd only gave me a 120 MB small image - probably the actual boot image on the DVD or something, while mkisofs gave me a full 4.2 GB image but a non-bootable one. How annoying!

By despair (or to be systematical), I also tried a tip I had up to that point considered too naive and fault-prone: to just cat the DVD-reader device to a file (i.e., 'cat /dev/sr0 > /tmp/recovery1.iso') and do you know what? It worked! VirtualBox booted the virtual host right of it and let me choose to do a factory reset (It is currently at 65%, so I don't know if it ultimately will work yet.)

Who would have thought that simple cat would be so true to the underlying bit device that it rendered the copied image as such a perfect copy? I am sure that given the right options to mkisofs, I could get the image bootable (it has a ton of different options to set this-and-that type of bootable disk to boot from a certain file, but I would rather not have to learn how to analyse the configuration of the DVD at hand enough to be able to pick all the right mkisofs options to make a one-to-one copy of it. Shouldn't mkisofs really be able to do that itself?

Video Woes

In McGuff's and Little's excellent "Body by Science", they advocate for strength-training with slow lifts (to remove any helping momentum) and to measure the Time Under Load rather than just number of repetitions, to get a more fine-grained control over your progress (or lack thereof). However, to concentrate on the weights and the clock at the same time is, of course, a drag. Better to film oneself and analyse the video after the workout - and which is the most ubiquitous movie camera around these days? Right, your mobile phone.

This scheme worked well for a number of workouts until the camera app in my phone crashed while recording, giving me a video no player will play. Why? Because the 3gp format of my phone (basically a mp4 variant with more compression) slaps on the frame index and codec information last in the file - and when the app crashes in mid-recording, no such index is added and the file ends up an inaccessible pile of junk data...

So, would I be forced to write-off the workout as a session without any data to track? Not without a fight. Some googling revealed that Federico Ponchio's Untrunc and Grau's Video Repair Software were the two most promising candidates. Both produced something from my broken video - unfortunately, Untrunc slapped on some codec all my players were missing (and that wasn't easily installable) but, luckily, Grau's program worked well, at least for the video - the audio become unsynced.

Alas, not only are the video and audio out of sync, it seems the video is somewhat in slow motion as well, so it turned out to be unusable for my post-workout analysis anyway. Too bad, but at least I got a story to tell out of it...

Fun with openwrt

I wanted to extend my network at home, with one router by the fibre-node downstairs and one by our desks upstairs. Also, I wanted to tryout DD-WRT. However, since there are two different versions of TP-Link WE1043ND floating around and I happened to get the black v2.1 version that are currently not supported by DD-WRT, I opted for OpenWRT instead.

It was quite scary to flash the new router with the OpenWRT image, but it worked flawlessly and in minutes I had configured it to do my bidding - despite OpenWRT looking seemingly overwhelming with all the choosable bells and whistles.

Scraping Fund Data of the Internet - an Orgy in Different Tools

Previously, I've used a combination of Bash and curl to scrape pages with fund statistics from investment sites and then Perl scripts to data-mine the scraped pages and analyse the data. Good so - small, customized, targeted Bash and Perl scripts.

However, recently, I wanted to also use the Norman value (an estimate on how much a fund will cost over 10 years of keeping it in a investment portfolio). The only source for these value was one popular fund comparison site, so I set out to bash and curl my way to a loop that would scrape the data off it - only to run into serious trouble...

Using the Firebug plugin in Mozilla Firefox, the built-in Developer's Tool in Google Chrome, and eventually resorting to Web-Charles to really be able to see what was going on, I was able to exactly pin-point what my browser sent the site when I navigated the pages - but I was unable to mock the request in my scripts. Unfortunately, the site relied just too much on Javascripts dynamically building form-data to be posted to the site with every request. So I had to think of something else.

Enter Selenium, the Web Automation Tool. Since it is using an ordinary browser to perform the surfing, dynamic Javascript mumbo-jumbo is no match for it. But I still wanted to script my usage and eventually choose Watir for the job, once again being surprised over how much joy it is to program in Ruby!

So, writing a short Ruby program using the Watir framework to operate on Selenium, I was able to get the data I wanted from the site in question.

However, I begun with restricting the funds to just those of the bank I am a customer off (it was my funds there that was most important to re-evaluate and perhaps exchange for others at the turn of the year). When I for kicks tried to scrape all of the 2 500++ funds of the site, I ran into new problems. The site was simply not stable enough. Now and then, one would either end up on the very first fund presentation page with a subtle "An error occurred"-message, or actually a full-fledged "An error occurred"-page, or one would seemingly be on the right track, getting to the next of the 100++ pages of 20 funds at a time, but then realizing that it had silently thrown you from the target tab back to the default tab. To battle this, I had to go heavy on error handling, turning my short and elegant Ruby script into a less nice collection of rescue blocks for this and that exception (some that I defined and raised myself, for example if browser.text.visible?("An error occurred") is true).

All in all, a quite educational and rewarding exercise.

Using ADSL and 3G concurrently

ADSL has its limitations but is, at least, ubiquitously available and comes with flat-rate. 3G subscriptions are usually only flat-rate up to a threshold of so and so many GB, over which you have to pay extra. But a cheap 3G subscription is a good backup against ADSL outages. Of course, both of them have the drawback of asymmetrically lower upstream bandwidths compared to the downstream speed.     

Thus, when you have a lot of data to upload (for instance after an extensive photo-session) you might very well want to concurrently upload on both types of network connections to keep the overall upload time down.

This is one recipe on how to achieve that (you will have to adopt to your own IP addresses and connection types below):

• Connect 3G dongle and initiate the connection on it
• Check your particulars:

$ ifconfig 

eth1      Link encap:Ethernet  HWaddr c8:0a:a9:a6:5f:c1  

          inet addr:192.168.1.150  Bcast:192.168.1.255  Mask:255.255.255.0

          inet6 addr: fe80::ca0a:a9ff:fea6:5fc1/64 Scope:Link

          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1

          RX packets:90315 errors:0 dropped:0 overruns:0 frame:0

          TX packets:51128 errors:0 dropped:0 overruns:0 carrier:0

          collisions:0 txqueuelen:1000 

          RX bytes:122726916 (122.7 MB)  TX bytes:5513301 (5.5 MB)

          Interrupt:42 

lo        Link encap:Local Loopback  

          inet addr:127.0.0.1  Mask:255.0.0.0

          inet6 addr: ::1/128 Scope:Host

          UP LOOPBACK RUNNING  MTU:16436  Metric:1

          RX packets:1356 errors:0 dropped:0 overruns:0 frame:0

          TX packets:1356 errors:0 dropped:0 overruns:0 carrier:0

          collisions:0 txqueuelen:0 

          RX bytes:239907 (239.9 KB)  TX bytes:239907 (239.9 KB)

ppp0      Link encap:Point-to-Point Protocol  

          inet addr:83.180.203.199  P-t-P:10.64.64.64  Mask:255.255.255.255

          UP POINTOPOINT RUNNING NOARP MULTICAST  MTU:1500  Metric:1

          RX packets:7 errors:0 dropped:0 overruns:0 frame:0

          TX packets:8 errors:0 dropped:0 overruns:0 carrier:0

          collisions:0 txqueuelen:3 

          RX bytes:130 (130.0 B)  TX bytes:181 (181.0 B)

• Add a routing table for the 3g: 

# sudo echo 2 3g >> /etc/iproute2/rt_tables

• Add to and fro routes for the 3G connection to the new table (my 3G gives me a ppp0 interface). Notice the priority of 1000, chosen to lie in between 0 and 32766:

# ip route add default via 83.180.203.199 dev ppp0 table 3g

# ip rule add from 83.180.203.199 table 3g prio 1000

• Check what you've got: 

$ ip route show table 3g

default via 83.180.203.199 dev ppp0 

•  And your complete routing rules:

$ ip rule

0: from all lookup local 

1000: from 83.180.203.199 lookup 3g 

32766: from all lookup main 

32767: from all lookup default 

•  Now you can use the 3G interface concurrently with your normal network (without packet in one direction getting confused and ending up where not intended). You can, for instance, verify it by ssh:ing out on the 3G interface with  ssh -b 83.180.203.199 <some machine> and on that machine, you can verify where you're actually coming from with, for instance, w. You probably don't have wired and mobile internet from the same provider, so it should be easy to tell them apart.
•  But what about the concurrent uploads, then? Well, I use  (in different terminals, of course):

$ rsync -e 'ssh -b 83.180.203.199' --progress -ha Pictures/2013-03* <remote-machine>:Pictures/

$ 'rsync -e 'ssh -b 192.168.1.150' --progress -ha Pictures/2013-02* <remote-machine>:Pictures/

Naturally, I would get better network utalization if I uploaded huge tar-balls of the content than rsync:ing perhaps many small files - however, I'm after rsync's capability to determine exactly what's new and needs to be uploaded and what's already there.

Also, a common use-case is, of course, to just upload the latest, massive folder of photos. Then you can use a little bash-fu to make each concurrent connection upload different files - for instance on the one the odd and on the other the even numbered photos:

$ rsync -e 'ssh -b 83.180.203.199' --progress -ha Pictures/2013-03-23/*{1,3,5,7,9}.jpg <remote-machine>:Pictures/2013-03-23/

$ 'rsync -e 'ssh -b 192.168.1.150' --progress -ha Pictures/2013-03-23/*{0,2,4,6,8}.jpg <remote-machine>:Pictures/2013-03-23/

Screen as a serial console software

How cool is that? Old, honest screen can actually be used to connect to a serial console. Typically, you nowadays have some fancy usb-to-serier converter and when you connect the usb, you'll probably find the right tty with

$ dmesg | grep ttyUSB

[1384486.485394] usb 3-1: FTDI USB Serial Device converter now attached to ttyUSB3

Then you can use screen to access the serial console (it's good to know the Baud rate, too):

$ screen /dev/ttyUSB3 115200

Neat!

(And, yeah, screen is old and the code-base is not maintained, so one should really migrate to tmux some day.)