When Ubuntu fails

I’ve been busy IRL so posting here has taken a backseat to other things, as well I haven’t had a lot of time to tinker. This is an old draft I had kicking around that I’ve cleaned up a bit.

Yes, I’m guilty of running for ‘weeks’ with a pending reboot required, this is probably not helping the situation.  I’ve probably also had several power fails etc with the system in a suspended state. Still, I didn’t expect my Ubuntu system to get to the state it did.

After rebooting my system, the system drive would no longer boot and I was dumped into the (initramfs) busybox

mount: mounting /dev on /root/dev failed: No such file or directory
mount: mounting /sys on /root/sys failed: No such file or directory
mount: mounting /proc on /root/proc failed: No such file or directory
Target Filesystem doesn't have /sbin/init.
No init found. Try passing init=bootarg.

BusyBox v1.10.2 (Ubuntu 1:1.10.2-2ubuntu7) built-in shell (ash)
Enter 'help' for a list of built-in commands.

(initramfs)

Ok I think, so there is some filesystem issue with my boot drive – booting a live CD version of Ubuntu should give me the tools to fix it. It turns out the answer was no – the live CD won’t help me either. Sigh, this is the type of thing I’d expect of Windows Vista but not Ubuntu.

Off the to forums and I turn up a post which shows others have had the same issue and the solution. From here it was a simple matter of booting the Ubuntu live CD to download a copy of SLAX to burn to CD then boot from the new SLAX CD to repair the ext4 filesystem. Good thing I had a Ubuntu live CD around.

Once your booted into SLAX, start a root shell and find the volume.

root@slax:~# fdisk -l

This will list all of the drives (if you have more than one) and the partitions on those drives. Next is simply a matter of issuing the filesystem check and repair command on the correct partition

root@slax:~# fsck /dev/sda2

You’ll want to say yes to fixing the problems obviously. Once this completes, simply reboot back to a working system.

Review: OCZ Vertex 3 120G SSD

I’m not entirely certain which event triggered my gear lust for a solid state drive (SSD), it was probably a mix of Jeff Atwood’s post, TechReport’s storage section, and the falling prices resulting in smaller SSDs down below the $100 price point. Whatever it was, I couldn’t really shake the idea of having a SSD in my work laptop – so I decided to get one.

Initially I had thought that a 60G-64G drive would fit the bill, being under the $100 price point and just big enough to hold the OS plus my Lotus Notes mail installation. After reviewing benchmarks, and reviews I decided to focus on the 120G size – in part due to a general recommendation that the 60G size is a bit small for most, and the benchmark numbers on the 120G are a bit better. The price was higher, but still within a very reasonable budget as SSDs are approaching $1 a Gig. The TechReport comparison of 120G-128G size helped me narrow my choice down to the OCZ Vertex 3.

While the Vertex 3 has been on the market a year, it still ranks as one of the fastest drives available. There were some issues with the SandForce SF-2881 controller, but firmware 2.15 is reported to be solid.

My laptop was running a 500G SATA2 Toshiba drive, configured as a single large partition running Windows 7. I had no interest in re-installing from scratch so my approach was to clone the working system onto the smaller drive. There are likely plenty of ways to do this, I was able to easily find a blog post describing how to do it – I roughly followed those steps but will document exactly what I did here.

Step 1) Reduce the partition on the big hard drive to be a bit less than the formatted capacity of the SSD. Initially after reading a bit I was hesitant to use GParted to do this as it seemed some folks had had problems with Windows 7 and GParted. Windows 7 also has a built in partition resize capability.

I ran into several issues trying to use the built in Windows 7 functionality. First up was some unmovable files causing issues. Even after turning off virtual memory and system restore, I still had issues. The Event Viewer was a help in identifying Chrome as holding onto some unmovable files, then I hit what I believe was an issue with NTFS Metafiles being unmovable and blocking my ability to shrink the partition smaller than 245G. At this point I threw my hands in the air and ran GParted from an Ubuntu Live USB key.

GParted ran to completion, but oddly gave me an error indicating something was wrong – but I couldn’t spot anything actually wrong. [Normally GParted should not give an error] The damage was done so I just rebooted and let Windows perform the necessary chkdsk activity. Things were fine, so either I mis-read that there was an error or it was something that was recoverable. Either way I was now happily running with a 100G partition.

Step 2) Use Clonezilla‘s “savepart” option to capture an image of the partition. Since I had a 500G drive which now had lots of empty space after the 100G system partition, I created a 2nd volume to store the captured image to. You can use a second USB mounted drive, or any number of other options including ssh with Clonezilla to store your image.

I will comment that Clonezilla is not for the timid, the user interface appear very complex and requires some careful reading to make sure you’re doing what you think you’re doing. Youtube has a number of walk throughs. For the 100G partition it took about 1:35 to backup.

Above you see the SSD attached to the ultra slim sled that the laptop hard disk was in, this is a very slim metal sleeve with a pull tab and some rubber bumpers. It fit nicely into my W520.

Step 3) Swap the drives. If you have a password on the drive, it’s a good idea to disable before removing it as USB enclosures and passworded drives don’t mix well. Install the new SSD, and place the existing drive into a USB enclosure. Boot the laptop into Ubuntu Live again and partition the new SSD drive, make sure to tag the new partition as with the ‘boot’ flag.

Step 4) Restore the image you saved with Clonezilla’s “restore part” option. In this case I was restoring from the 2nd partition on the original hard drive that is now mounted as a USB volume. Clonezilla warns you twice when restoring a partition to validate you’ve got the correct destination, a nice paranoid touch.

The restore ran nearly 3x faster taking about 37 minutes.

Step 5) Boot into windows, chkdsk may have run again but with the SSD it seemed to take no time at all. You might want to visit the OCZ site and grab the toolbox utility to validate you’ve got the latest firmware, I did this to verify I had 2.15.

Performance

After I did the clone, I ran some boot time tests on the hard drive. I tested immediately after I had completed step 5 with the SSD. For work I need Lotus Notes up and running to access my calendar etc, so that was a logical pattern to benchmark – how long to get back to key information? I used a stop watch, and the times include the time I spent typing in the two passwords and navigating to the icon to launch Notes. It’s not terribly scientific, but I think the results still speak for themselves.

Disk test 1 Disk test 2 Disk test 3 SSD test 1 SSD test 2 SSD test 3
Cold boot to Windows login 1:22 1:24 55 23 23 23
Login to launch of Notes 1:42 1:13 1:44 10 10 10
Lotus Notes ready 40 44 40 10 10 11
Total time 3:45 3:22 3:19 43 43 44

This is crazy hot – more than 3x faster, under a minute from a cold boot.

Now certain operations don’t seem any faster. Resuming from hibernation feels to be about the same speed. This makes sense as the performance difference for sequential reads isn’t much different. It seems in normal usage, lots of little things are more immediate too. Some of this is likely simply moving from a SATA2 to a SATA3 drive, but I’m convinced no spinning platter could keep up with the SSD.

How To: DD-WRT on the Netgear WNR3500L

I’ve been a fan of DD-WRT for some time, a Linksys WRT54GL is the heart of my home network. While I don’t have an immediate need for a new router, I’ve been thinking for some time that I should pick up a reasonable backup in case the WRT54GL fails. There are also some really interesting projects you can do with a modified router, and having a spare will let me tinker with some of those.

I came across a good deal on a refurbished Netgear WNR3500L, it was about half the price of a new unit. This was too tempting: N networking (I don’t really need it, but why not stay current?), gigabit wired networking (a must have), USB port (I wanted this for expansion options), but the feature that really sold me was 64MB RAM and 8MB ROM – more than enough for DD-WRT with all the features.

This particular router is also supported by Tomato another very popular firmware. There is also a somewhat suspect Netgear sponsored open firmware, the site seems to have instructions for installing an older version of Tomato – personally I’d steer clear of this. Unfortunately the other popular router firmware OpenWRT doesn’t currently support this router. Of course, I prefer DD-WRT which does support this router.

One of the reasons that I blog is to have a record of what I did, far too often I end up needing to do something again, or refer back when I’m doing an upgrade. Another reason is to help me get organized to do some of these things – finding the instructions is usually easy, understanding them and what you’re about to do takes some time. I probably spent 2hrs reading through forums and searching various things before I leapt in to flash the router, I strongly recommend you do the same. Also this write-up skips over some of the frustrating head scratching I do when things don’t work, that’s part of the learning process.

There is a v2 of this router, and v2 is NOT supported by DD-WRT. Apparently the boxes they come in are identical, but the router itself has a v2 marking. I did find a wiki that documents the v1 and the v2, I was able to match the FCC numbers listed to confirm I had a v1. For what it’s worth, my stock firmware was V1.0.2.50_31.1.25NA.

The DD-WRT site has some excellent instructions on doing the installation, I’ll avoid repeating those but will document the steps I took. I will repeat some of the cautions from the site:

  • The WNR3500L requires a NEWD-2, K2.6 build of DD-WRT
  • You will first need to upload a special .chk build (also K2.6) of DD-WRT to the router to go from Netgear’s factory firmware. Then you may install a .bin build of DD-WRT.
  • Spend some time browsing the DD-WRT Forums. There’s a lot of chaotic information to ingest.
  • Read the peacock thread.

Before we get started, let’s talk about the 30/30/30 reset. Yes, it’s over the top and very conservative. Weigh those 90 seconds against the potential problems you might run into trying to by rushing through this, sure building a JTAG connector to resurrect a bricked router might be interesting but it’s going to take you a lot more than 90 seconds.

The following steps should not be considered an alternate installation procedure, please use the official DD-WRT instructions – consider this a walk through of the official instructions:

  1. Power on the device. Give it a minute to finish booting, the front panel LEDs should stabilize.
  2. Plug in your linux box, it’ll automatically get an IP address (192.168.1.2)
  3. Perform a 30/30/30 reset
    a) With unit powered on, press and hold the reset button for 30s
    b) Without releasing the reset button, unplug the router, wait 30s
    c) Still holding the reset button, plug the router in and wait 30s
  4. Now wait for the PC to get an IP address assigned to it, this may take a minute or two be patient.
  5. As I’m using Linux Firefox will be the browser of choice
    Connect to 192.168.1.1
    user: admin
    password: password
    (the default user and password)
  6. Since the router isn’t connected to anything except my PC the automatic upgrade process will fail due to no internet connection, that’s fine.
  7. Click on “Router Upgrade” in the menu on the left side.
  8. Choose the .chk file you downloaded as per the DD-WRT site
  9. You will get a warning: “Warning! You are trying to download the firmware which the region is different from the current firmware you had.” – this is ok, continue.
  10. Wait for the upload process to complete.
    Once the upload is done, it will automatically move on to updating settings.
    A minute or so more it will show a DD-WRT screen.
  11. Start a timer for 5 minutes and wait. Yes, likely paranoid as we could have probably started the timer when we started the upload process, but again this isn’t something you really want to rush. Time for coffee.
  12. At this point we’re running DD-WRT, just not the version we want.
  13. Do another 30/30/30 reset
  14. It took my system about 2 minutes to get an IP address (192.168.1.101)
  15. DD-WRT will force us to set a username and password, for now I suggest the same as stock.
  16. Click on the Administration tab, followed by the Firmware Upgrade sub-tab.
  17. I set ‘After flashing, reset to’ ->  “Reset to Default settings”
    Pick the big file you downloaded as per the DD-WRT site.
  18. It will flip to a reset screen once the firmware has been installed
  19. Wait 5 minutes. If you’re really paranoid perform another 30/30/30 reset after that.
  20. Done. Configure your new router.

I primarily use Chrome, but have experienced problems with DD-WRT and Chrome in the past. It turns out this time that FireFox was unable to configure the router (but it did handle the firmware install just fine). Consider this a caution, if you’re having trouble with this process it might be your browser.