Algorithmic Pricing – Bad for Everyone

There is so much promise that technology has, but it feels like we’ve taken the wrong turns somewhere along the line and now we’re in a very weird place. While this post will focus on ancestry.ca – the intent is to comment on the wider problem, not this particular website.

From a family tree / genealogy point of view – having computer access to mountains of data makes searching for information so much easier than it used to be. Now, there is also something satisfying about traveling to a given location and finding the original paper records, or walking an old graveyard and finding tombstones that tell a story about the people who lived in that community. Ideally you want to combine all of these approaches to have a rich experience about exploring a family history.

While you can use ancestry.ca for free, the real value is access to the trove of data they have available. To get access you pay a monthly fee – and this is fair – running a website (even this one) isn’t free. But what do you charge people? It’s pretty variable, it depends on some fix costs – but there are many other variable costs, and coming up with a fair number is hard.

Of course with marketing you also want to set a fixed price, but offer deals to encourage customers to make the leap from fee to paying. Here is where I struggle with the approach when those discounts move around dynamically.

A gift membership seems like a great way to get someone started. However, this is what you get if you want to go down that path

Two plans, and either 6 or 12 months.

Contrast this with what just subscribing offers you

Ok, well – there is a lower plan that is significantly cheaper if we consider a 12 month plan. So maybe a good gift is that “All Canadian Records” version. That was my plan – and so I printed out the website page and said – hey, here you are I will gift you the 12 month lowest tier plan.

Then weirdness started. The first attempt to take the logged in ancestry account and subscribe landed at a page that offered up a different price.

Wait, what? Where did my 12 month option go? This is not nearly as good a deal. Let’s grab another computer and visit the site to see what’s going on.

Great, now we only see a monthly price? No more multi-month discounts?

Of course, the next thing I do is start trying different browsers. I get a variety of the 3 possible choices all captured above. However, I’m still struggling to figure out how do I purchase the right “cheap” one of $119.88 / year?

In the end, I kept creating new incognito windows in Chrome until I got the right price. This actually took a bit of trying as mostly I was getting only the 6 month option. Once I got the right price I clicked on the “Become a Member” button, and signed in on that window. As I worked through the credit card payment page, it seemed – yes, I was paying for the right $119.88 price – my total was +tax, and it went through on my credit card at that price. Problem solved.

This does make me wonder, how would a normal person do this? A gift membership would push people to a higher level plan (and extra $60/year). If they got only the monthly cost ($14.99) that stacks up to an $60.11/year. The 6-month plan is an extra $20.10/year. From a yearly price point – the cheapest gift membership is 50% more expensive; monthly payments also work out to 50% more expensive, and the 6-month option is 17% more expensive.

Just like casinos, the house always wins with algorithmic pricing. This type of pricing makes me less (and likely many others) less likely to actually buy things from companies that do this. I value the service they are providing, but I want to pay a fair price – not one that is a roll of the dice.

Adventures in 4K – Ripping 4K UHD Blu-Ray

For my birthday a few months ago, I got a copy of The Matrix in 4k. Previously I had only the original DVD that I bought when it first came out. I popped the 4k blu-ray into my blu-ray drive and started up MakeMKV only to discover that my system was unable to read a UHD disc.

Thankfully the 4k blu-ray comes with a normal blu-ray that contains a 1080p copy of the movie and I was able to rip that to my personal collection. While I do have a 4k capable TV, my primary projection setup is still only 1080p so having more bits available isn’t actually a better setup. Still, owning a 4k disc and not being able to use it bugged me.

It turns out the MakeMKV folks run a forum, and there are recommendations there for the right drives to buy in order to rip the 4k discs. There is a thread Ultimate UHD Drives Flashing Guide Updated 2024 which is required reading if you want to get started. I also checked out CanadaComputers which is my local go-to computer store, often having better prices than you can find online.

My pick was the LG WH16NS40 which was both low cost, and appeared to be well supported by the MakeMKV forum. Of course, it isn’t as simple as buy the drive and rip 4k media, you need to modify the firmware. The fact that I had to modify the drive to get it to do what I wanted made this a must have item so it went on my Christmas list. Thankfully I was on the good list and when it was time to unwrap gifts I had my hands on a new drive.

Installing the drive into my Linux machine was pretty straight forward. I ended up replacing another older DVD drive I had in there. On the label of my new drive, I could see the model number (WH6NS40) and manufacture date (June 2024). There was also an indication of the ROM version (1.05).

I run MakeMKV in a container, for me this is a great way to encapsulate the right setup and make it easy to repeat. The new drive showed up just fine to MakeMKV – but I didn’t expect it to support 4k UHD discs just yet.

I will summarize things further down, so you can skip to the summary if you want. However, the bulk of this post will be my discovery process on re-flashing the drive.

Time to head off to the guide and read it carefully.

The first thing I took note of was the correct firmware I wanted based on my drive. This was in the “Recommandation” section near the top.

WH16NS40 on any Firmware directly to > WH16NS60 1.02MK

So I want the 1.02MK version, and it seems I can get there with a single flash vs. needing to do multiple steps.

A bit further down in the same guide, I came across

LG 1.04+ / BU40N 1.03 / Asus 3.10+ and similar
The newer OEM firmwares cannot be flashed easily due to the additional downgrade checks implemented by the drive/firmware manufacturer.

Oh oh. So I may have problems? I am pretty sure I have the 1.05 firmware.

As I read on, it seems the recommended flashing tool is for Windows, and while I have a few Windows systems the drive is installed in a box that only has Linux on it. I spent some time reading through various forum posts and searching for other related material.

At this point I have more confidence that yes, my drive is supported – but it’s a question about how exactly to fix this drive (under Linux) to make it go. Bonus points if I manage to do this all inside of a container.

I did find an older thread that discusses flashing things under Linux. It pointed at a stand alone flashing tool on github, but it was reading through this thread when I discovered that MakeMKV itself contains the sdftool and supports the flashing process. This means I already have the tool inside the MakeMKV container.

Here is how I run the container

For your system you will want to adjust the volume mappings and device mappings to match what is on your host system. This works great for me, and I can access both of the blu-ray drives on my system and write newly ripped files to my host filesystem.

Looking at the browser view of the MakeMKV container I can see that the new drive is recognized, and in the right side panel it even calls out the details for LibreDrive support.

Shelling into the docker container, I can see that sdftool is a link to makemkvcon.

I had read about the possibility of dumping the original firmware as a backup plan in case things go very badly, but it seems this is actually not possible. It seems the manufacturers have made this more complicated in the name of security or something.

I grabbed the “all you need firmware pack” from the guide. This is a very small set of alternative firmwares, only one matches my LG ‘desktop’ sized drive so it was easy to identify the one I wanted to use.

I also needed the SDF.bin file that is hosted on the makemkv site.

In theory, I have all the bits I need. The sdftool, the SDF.bin, and the modified firmware.

At this point, I’m back following the guide. The Mac/Linux portion which walks you through things. I can dump information about the current firmware from my drive

Now I know the existing firmware version, it does not appear to be an exact match to the ones in the list from the guide under “Newer OEM Firmwares and encrypted”. However, the following is a pretty close match:

This drive was made in June 2024 and most probably has a firmware from after 2020 – so a very close match to the list above, and the date of manufacture makes it very likely that my drive has ‘encrypted’ firmware.

Ok – to recap what the plan looks like.

  1. Grab the sdf.bin file
  2. Download the modified firmware(s)
  3. Dump existing firmware versions – determine if you are encrypted or not (likely you are)
  4. Flash the drive

Easy right?

From outside the container we can copy in the firmware we need

And inside the container we can pull down the SDF.bin file.

Then we just need to do the very scary flash part.

There is a very long (minutes) pause where the flashing is taking place.. longer than I can hold my breath.. uh.. did I just make a brick?? fuuuuuu….

I can see from another terminal session that it is eating CPU, pegged at 100%.

After 10+ mins of hanging.. I hesitantly CTRL-C’d the thing..

Thankfully, everything seems ok – I’m exactly where I started. Whew.

I found that adding the verbose (-v) flag was probably a good idea, and a forum thread that indicated that there should be more output from the command. Maybe it’s getting stuck starting up?

I had a few thoughts. Maybe I need to run the container with less restrictions? (docker –privileged) No, that didn’t change anything.

Then I found someone having the same problem recently. It seems the solution they used was to just use Windows. I did ponder how I might setup a temporary Windows install to do this. Then I found this thread that discusses MakeMKV hanging after loading the SDF.bin file, this feels like it may be the same problem. In that case the issue is with the most recent version of MakeMKV (1.17.8).

I started looking for an older version of the container I’ve been using, one that has MakeMKV (1.17.7). It turns out that jlesage/makemkv:v24.07.1 is a few tags back, but has that version. Let’s see if using this version will work better.

This seems to be much better, I’m now getting an error message instead of a 100% CPU hang. Also, apparently I need to remove the disc from the drive which is something I can do.

I only very briefly held my breath as I typed in ‘yes’ and let it continue to do the work. It only took a minute or so to flash the drive and report success.

I needed to re-start the makemkv container. Then it was showing me my drive was good to go

As you can see the LibreDrive information now shows

And I can now read 4k UHD blu-ray discs without problem. I was able to rip the 4k version of the Matrix (53Gb) without issue. My setup was only showing [4x] speed, but I suspect this is more a limitation of my overall system vs. the drive which I suspect can go faster. I’m still very pleased to be able to pull the bits.

Summary – the TL;DR version

Recent versions of the LG WH16NS40 can be modified to read 4k UHD blu-ray discs. This can be accomplished under Linux, using the MakeMKV container.

There is a bug in MakeMKV version 1.17.8 which causes it to hang with 100% CPU. Using version 1.17.7 still works as of the date of this post.

Absolutely read the guide.

Start up the MakeMKV container.

I downloaded the firmware bundle, and picked the matching one for my drive. I then copied it from my host filesystem into the container

Then shell into the container and download the SDF.bin file.

Now we issue the flash command

That’s it. We have a modified firmware installed.  Time to enjoy 4k goodness.

OpenBeken with Globe Electric Smart Plugs

I needed some more home automation friendly outlets, and my preferred Sonoff S31 outlets are hard to get at the moment. It also seems that the S40 has been released as a replacement, but moves away from the ESP micro-controller. This got me looking for what I could get locally.

The Home Depot carries the Defiant series, which when I looked up the details on the FCC website it seemed like a great candidate having an ESP32 inside. However, it seems that the eFuse has been flipped to prevent any further firmware updates. I then moved on and found a sale price of $22.99 for a 3 pack of the Globe Electric Smart Plugs.

Now these plugs do not have an ESP device in them, but a WB2S module. Some folks take advantage of the fact that this is pin compatible with an ESP device and will just swap in a whole new logic board. However, you can also use OpenBeken which I’ve done before. Since these were pretty cheap I picked up 3 boxes (9 plugs).

These plugs are fairly compact and you can fit two in an outlet (top and bottom) if you wanted.

The first thing we need to do is open these. This is fairly easily done using a pocket knife and a little bit of patience / prying. After doing a few of these I got pretty quick, a minute or two to open them. I captured a short video I put up on YouTube showing this process.

Now that it’s open, you can remove the exterior case entirely and you have access to the internals. The micro-controller was easy to spot, it is on it’s own tiny circuit board.

You can see why some folks are simply removing the entire controller board and swapping in a compatible one with a more friendly chip to program. However, with OpenBeken we have the option of reprogramming this chip with some firmware to give us local only control (via Home Assistant).

At the top you can see this is a WB2S board, and the chip is a BK7231TQN32. For the most part this chip is referred to as a BK7231T.

We’re going to need to hook up 4 wires to this to reprogram it, 3.3V, GND, Tx and Rx. Let’s figure out where those pins are.

This is the other side of the board, and if you look carefully you can see at the bottom edge we have starting from the left side BAT, GND, IRX, ITX. These are the 4 pins we need to connect with. Thankfully we can access each of these from the bottom side of the board as the controller board slots into the main circuit board on it’s edge.

Not very pretty, but it gets the job done. The wires are from some stripped ethernet cable and I think they are 26AWG – fairly small. While this was convenient for me to connect some wires to, upon inspection it seems the only reason there is any solder here is to support the board-to-board connection, I’m pretty sure all of the actual interfacing to the circuits is on the other side.

Someone on the Elektroda forum seems to have done the first work on one of these plugs. A lot of people using OpenBeken have used the CloudCutter project to reprogram things without wiring anything up, but this seemed more complicated to me and not guaranteed to work with all devices.

My first attempt I used the same software as I had previously. The only change was to download a different firmware because this is a different module. Unfortunately I goofed somewhere and when flashing the device it failed, then it was no longer responsive to my attempts to connect via serial at all. It may be a brick now, boo.

For my second attempt, I did more reading about the ESPHome support for BK72xx chips. There is quite detailed information about flashing the chips, and it seems to strongly recommend you do not use the old tool I had previously used. There is also a section there on un-bricking things which I’ll have to try later.

Thus I picked the ltchiptool – and installed it under Windows. The easy way to do this on Windows was use the Microsoft Store to install “Python 3.10” – then run pip install ltchiptool[gui]

Once installed, you can launch the GUI with python -m ltchiptool gui

While I may eventually end up using ESPHome, I decided to stick with OpenBeken and grabbed the latest OpenBK7231T_UA_1.17.800.bin from their download page. I first used the ltchiptool to download the existing flash image to confirm my wiring was working. Then I held my breath and uploaded the firmware.. and it worked.

Then it was a simple matter of repeating things. Crack open the case. Solder some wires. Hook it up to ltchiptool, download to confirm things are working, upload new firmware. De-solder the wires. Re-assemble. Now I had 8 outlets reprogrammed.

A dab of crazy glue helped re-seal the exterior cases. So far this seems just fine after a few insert / removals of the plugs.

Next we need to do some configuration. Upon first boot the device will offer up a captive portal. Connect your computer to the “OpenBK76231N_XXXXX” WiFi and visit http://192.168.4.1 in a browser. From here we can pick “Config” then “Configure Wifi & Web” to setup the Wifi connection.

Once the device is on our WiFi network, it can then see the internet which allows us to use the “Web Application” to further configure the device. You can launch the Web Application from the main landing page. I did find that it was important to access the device via IP address (ie: http://192.168.1.56) vs. by name.

Once we launch the Web Application, we can easily configure the device by using the “Config” tab, and searching for the right template from the web. I have to say this is a pretty clever way to do things.

You can see I’ve searched up the “Globe Globe smart plug” and have used “Copy Device Settings” to populate the “Pin Settings”. At the bottom of the page there is a “Save” button to persist the choice. Now if we revisit the Home page we will see the device now offers up control of the power. The button on the side of the plug also now works to toggle power.

The last thing we’ll do is connect this up to Home Assistant. The easy path here is to setup MQTT (Config->Configure MQTT) and get connect to your broker. I was left scratching my head why Home Assistant wouldn’t see these new devices – and after a lot of probing around my MQTT setup and verifying that data was flowing, I finally watched the video the explains it.

Short story was – after you’ve got MQTT setup, and the main page will show you the MQTT State is connected – you need to do one more thing to have it emit a discovery payload to Home Assistant. (Config->Home Assistant Configuration) and hit “Start Home Assistant Discovery”. Then the devices just appeared with their short name in Home Assistant.