Author: Roo

Archiving Floppies

I’m slowly getting to clearing out some of the old office stuff at home, and yes, I appear to still have some 3.5″ floppies. I did in fact have a 3.5″ floppy drive, but it was in an old husk of a former PC. My desktop machine has a modern power supply and didn’t even have the right power connector to hook up the drive (easy fix with an adapter) – at least the motherboard still had the right connector to hook up the data cable.

I then had to do the right BIOS dance to actually enable the device, once this was done I could see it under Linux as /dev/fd0. Unfortunately the handful of disks I tried to mount gave errors, it seemed either this drive is faulty or all of my disks are expired. Now, these are floppies from the early 90’s – which is oh my 35 years ago!

Time to bust out ddrescue, and see if I can image any of these disks to pull data. Sadly my initial attempts were not great – I wasn’t getting much data off of these at all. Maybe this is a huge waste of time. I found the useful seeming ddrescueview which gives me a way to look at the status of the rescue attempt.

Let’s cover the basics. My initial attempts looked like

This worked, but I got a lot of errors. Adding the -d flag seemed to help a lot, but later I found out that I needed more flags to make this right.

I found a useful wikipage entry from the archiveteam specific to recovering floppies.

Here is the ddrescueview visualization of my initial attempt:

So not great. Next up is when I added the -d flag

Better. Of course as I decided to make sure this was repeatable, I tried removing the -d flag and running it again to make sure it was really bad. This time I got a completely clean read (fully green). There were 2 errors reported, but it retried and it was good?

So I start trying various combinations to see if I’m getting repeatable results. Overall it’s random errors and no clean reads again.

Now the clever thing that ddrescue does, is maintain a map file. This captures what was done, and allows you to run another pass to try to have more luck. This is what I need. Referencing the archiveteam advice I landed on this as the right combination

Let’s break down the flags

  • -d : direct access
  • -b512 : sector size of 512 bytes, important for direct access
  • -r 3 : retry errors 3 times
  • –retrim : allows us to re-run, and re-try failed blocks in the map

Using this magic, I was able to run the command a second time and get a clean read! So you can either be lucky, or use the map file and try a few times with the right settings.

Now I can mount the image under linux

This particular floppy apparently contained a few rescue tools (NDD.exe ring any bells) Well, glad I got those bits back – guess I’ll toss it on the pile and move on to the next one.

Now that I have things sorted out – I’m finding a couple that read clean, which is pretty cool given the files are from 1991. Amazing how little fits on these floppies, when it used to seem like so much.

I did manage to ‘crash’ the floppy drive with bad disks or something, because it would get into a state that rebooting the machine would not fix. Powering it off for a minute or two and a full cold boot seemed to get things back on track. When it was busted I’d get errors like:

I did run into more problems just like this and I really don’t understand what was wrong, or how to get it to behave again. Very frustrating. I just had to keep trying cold boots and different floppies. Looking at dmesg I see:

I picked up a used USB floppy drive locally, it was only $15 and it gave me a secondary device to try some of these floppies with — and I was hitting my head against the wall with the errors above.

The USB floppy appears on my system as a drive /dev/sdc – but I can just use that device in place of /dev/fd0 and the same commands work. Hopefully resetting the state will be easier as I can just unplug the USB drive and try it again. We’ll see if it gets into a similar busted state (which appears to be triggered by bad reads). So far it seems much more stable overall and I’m working my way through my old floppies.

The USB floppy drive worked really well. It is starting to seem like that old 3.5 floppy drive I installed in my machine was maybe not so stable. Some floppies that had many errors, read just fine with the USB floppy drive.

To speed things up, I adopted a two phased approach. Trying an optimistic version which would fail out quickly – followed by the more aggressive 3 retry version above if I determined I wanted to get as much data as possible. This is the quick version:

As a bonus for anyone who’s hung on this far into the post, let me share some of the output you get from the ddrescue tool showing the progress it makes:

You can see above, that it did in fact get to 100%, but slowly and required a secondary run to finish.

This was certainly a trip down memory lane, I’m glad I persisted in trying to read the data. There were a few files I wanted to keep out of the pile of floppies, and now I’ve got the archived with my other files to keep.

Algorithmic Pricing – Bad for Everyone

There is so much promise that technology has, but it feels like we’ve taken the wrong turns somewhere along the line and now we’re in a very weird place. While this post will focus on ancestry.ca – the intent is to comment on the wider problem, not this particular website.

From a family tree / genealogy point of view – having computer access to mountains of data makes searching for information so much easier than it used to be. Now, there is also something satisfying about traveling to a given location and finding the original paper records, or walking an old graveyard and finding tombstones that tell a story about the people who lived in that community. Ideally you want to combine all of these approaches to have a rich experience about exploring a family history.

While you can use ancestry.ca for free, the real value is access to the trove of data they have available. To get access you pay a monthly fee – and this is fair – running a website (even this one) isn’t free. But what do you charge people? It’s pretty variable, it depends on some fix costs – but there are many other variable costs, and coming up with a fair number is hard.

Of course with marketing you also want to set a fixed price, but offer deals to encourage customers to make the leap from fee to paying. Here is where I struggle with the approach when those discounts move around dynamically.

A gift membership seems like a great way to get someone started. However, this is what you get if you want to go down that path

Two plans, and either 6 or 12 months.

Contrast this with what just subscribing offers you

Ok, well – there is a lower plan that is significantly cheaper if we consider a 12 month plan. So maybe a good gift is that “All Canadian Records” version. That was my plan – and so I printed out the website page and said – hey, here you are I will gift you the 12 month lowest tier plan.

Then weirdness started. The first attempt to take the logged in ancestry account and subscribe landed at a page that offered up a different price.

Wait, what? Where did my 12 month option go? This is not nearly as good a deal. Let’s grab another computer and visit the site to see what’s going on.

Great, now we only see a monthly price? No more multi-month discounts?

Of course, the next thing I do is start trying different browsers. I get a variety of the 3 possible choices all captured above. However, I’m still struggling to figure out how do I purchase the right “cheap” one of $119.88 / year?

In the end, I kept creating new incognito windows in Chrome until I got the right price. This actually took a bit of trying as mostly I was getting only the 6 month option. Once I got the right price I clicked on the “Become a Member” button, and signed in on that window. As I worked through the credit card payment page, it seemed – yes, I was paying for the right $119.88 price – my total was +tax, and it went through on my credit card at that price. Problem solved.

This does make me wonder, how would a normal person do this? A gift membership would push people to a higher level plan (and extra $60/year). If they got only the monthly cost ($14.99) that stacks up to an $60.11/year. The 6-month plan is an extra $20.10/year. From a yearly price point – the cheapest gift membership is 50% more expensive; monthly payments also work out to 50% more expensive, and the 6-month option is 17% more expensive.

Just like casinos, the house always wins with algorithmic pricing. This type of pricing makes me less (and likely many others) less likely to actually buy things from companies that do this. I value the service they are providing, but I want to pay a fair price – not one that is a roll of the dice.

Adventures in 4K – Ripping 4K UHD Blu-Ray

For my birthday a few months ago, I got a copy of The Matrix in 4k. Previously I had only the original DVD that I bought when it first came out. I popped the 4k blu-ray into my blu-ray drive and started up MakeMKV only to discover that my system was unable to read a UHD disc.

Thankfully the 4k blu-ray comes with a normal blu-ray that contains a 1080p copy of the movie and I was able to rip that to my personal collection. While I do have a 4k capable TV, my primary projection setup is still only 1080p so having more bits available isn’t actually a better setup. Still, owning a 4k disc and not being able to use it bugged me.

It turns out the MakeMKV folks run a forum, and there are recommendations there for the right drives to buy in order to rip the 4k discs. There is a thread Ultimate UHD Drives Flashing Guide Updated 2024 which is required reading if you want to get started. I also checked out CanadaComputers which is my local go-to computer store, often having better prices than you can find online.

My pick was the LG WH16NS40 which was both low cost, and appeared to be well supported by the MakeMKV forum. Of course, it isn’t as simple as buy the drive and rip 4k media, you need to modify the firmware. The fact that I had to modify the drive to get it to do what I wanted made this a must have item so it went on my Christmas list. Thankfully I was on the good list and when it was time to unwrap gifts I had my hands on a new drive.

Installing the drive into my Linux machine was pretty straight forward. I ended up replacing another older DVD drive I had in there. On the label of my new drive, I could see the model number (WH6NS40) and manufacture date (June 2024). There was also an indication of the ROM version (1.05).

I run MakeMKV in a container, for me this is a great way to encapsulate the right setup and make it easy to repeat. The new drive showed up just fine to MakeMKV – but I didn’t expect it to support 4k UHD discs just yet.

I will summarize things further down, so you can skip to the summary if you want. However, the bulk of this post will be my discovery process on re-flashing the drive.

Time to head off to the guide and read it carefully.

The first thing I took note of was the correct firmware I wanted based on my drive. This was in the “Recommandation” section near the top.

WH16NS40 on any Firmware directly to > WH16NS60 1.02MK

So I want the 1.02MK version, and it seems I can get there with a single flash vs. needing to do multiple steps.

A bit further down in the same guide, I came across

LG 1.04+ / BU40N 1.03 / Asus 3.10+ and similar
The newer OEM firmwares cannot be flashed easily due to the additional downgrade checks implemented by the drive/firmware manufacturer.

Oh oh. So I may have problems? I am pretty sure I have the 1.05 firmware.

As I read on, it seems the recommended flashing tool is for Windows, and while I have a few Windows systems the drive is installed in a box that only has Linux on it. I spent some time reading through various forum posts and searching for other related material.

At this point I have more confidence that yes, my drive is supported – but it’s a question about how exactly to fix this drive (under Linux) to make it go. Bonus points if I manage to do this all inside of a container.

I did find an older thread that discusses flashing things under Linux. It pointed at a stand alone flashing tool on github, but it was reading through this thread when I discovered that MakeMKV itself contains the sdftool and supports the flashing process. This means I already have the tool inside the MakeMKV container.

Here is how I run the container

For your system you will want to adjust the volume mappings and device mappings to match what is on your host system. This works great for me, and I can access both of the blu-ray drives on my system and write newly ripped files to my host filesystem.

Looking at the browser view of the MakeMKV container I can see that the new drive is recognized, and in the right side panel it even calls out the details for LibreDrive support.

Shelling into the docker container, I can see that sdftool is a link to makemkvcon.

I had read about the possibility of dumping the original firmware as a backup plan in case things go very badly, but it seems this is actually not possible. It seems the manufacturers have made this more complicated in the name of security or something.

I grabbed the “all you need firmware pack” from the guide. This is a very small set of alternative firmwares, only one matches my LG ‘desktop’ sized drive so it was easy to identify the one I wanted to use.

I also needed the SDF.bin file that is hosted on the makemkv site.

In theory, I have all the bits I need. The sdftool, the SDF.bin, and the modified firmware.

At this point, I’m back following the guide. The Mac/Linux portion which walks you through things. I can dump information about the current firmware from my drive

Now I know the existing firmware version, it does not appear to be an exact match to the ones in the list from the guide under “Newer OEM Firmwares and encrypted”. However, the following is a pretty close match:

This drive was made in June 2024 and most probably has a firmware from after 2020 – so a very close match to the list above, and the date of manufacture makes it very likely that my drive has ‘encrypted’ firmware.

Ok – to recap what the plan looks like.

  1. Grab the sdf.bin file
  2. Download the modified firmware(s)
  3. Dump existing firmware versions – determine if you are encrypted or not (likely you are)
  4. Flash the drive

Easy right?

From outside the container we can copy in the firmware we need

And inside the container we can pull down the SDF.bin file.

Then we just need to do the very scary flash part.

There is a very long (minutes) pause where the flashing is taking place.. longer than I can hold my breath.. uh.. did I just make a brick?? fuuuuuu….

I can see from another terminal session that it is eating CPU, pegged at 100%.

After 10+ mins of hanging.. I hesitantly CTRL-C’d the thing..

Thankfully, everything seems ok – I’m exactly where I started. Whew.

I found that adding the verbose (-v) flag was probably a good idea, and a forum thread that indicated that there should be more output from the command. Maybe it’s getting stuck starting up?

I had a few thoughts. Maybe I need to run the container with less restrictions? (docker –privileged) No, that didn’t change anything.

Then I found someone having the same problem recently. It seems the solution they used was to just use Windows. I did ponder how I might setup a temporary Windows install to do this. Then I found this thread that discusses MakeMKV hanging after loading the SDF.bin file, this feels like it may be the same problem. In that case the issue is with the most recent version of MakeMKV (1.17.8).

I started looking for an older version of the container I’ve been using, one that has MakeMKV (1.17.7). It turns out that jlesage/makemkv:v24.07.1 is a few tags back, but has that version. Let’s see if using this version will work better.

This seems to be much better, I’m now getting an error message instead of a 100% CPU hang. Also, apparently I need to remove the disc from the drive which is something I can do.

I only very briefly held my breath as I typed in ‘yes’ and let it continue to do the work. It only took a minute or so to flash the drive and report success.

I needed to re-start the makemkv container. Then it was showing me my drive was good to go

As you can see the LibreDrive information now shows

And I can now read 4k UHD blu-ray discs without problem. I was able to rip the 4k version of the Matrix (53Gb) without issue. My setup was only showing [4x] speed, but I suspect this is more a limitation of my overall system vs. the drive which I suspect can go faster. I’m still very pleased to be able to pull the bits.

Summary – the TL;DR version

Recent versions of the LG WH16NS40 can be modified to read 4k UHD blu-ray discs. This can be accomplished under Linux, using the MakeMKV container.

There is a bug in MakeMKV version 1.17.8 which causes it to hang with 100% CPU. Using version 1.17.7 still works as of the date of this post.

Absolutely read the guide.

Start up the MakeMKV container.

I downloaded the firmware bundle, and picked the matching one for my drive. I then copied it from my host filesystem into the container

Then shell into the container and download the SDF.bin file.

Now we issue the flash command

That’s it. We have a modified firmware installed.  Time to enjoy 4k goodness.