Correct Playlist – Segment Map for Bluray

Before streaming was the primary way to get new content, I collected a lot of movies on DVD / bluray. I still have a large collection, but have been slowly converting it to be hosted on my Plex server. This gives me a Netflix like viewing experience, but for my own personal movie collection.

Handbrake is great for dealing with DVDs. MakeMKV is how I rip bluray disks, then feed the resulting rip into Handbrake to compress it down. I do all of this on my Ubuntu system.

Generally this works really well. Handbrake will automatically select the longest video, and that’s almost always the actual movie itself. With MakeMKV that selection is more manual, but picking 1 title from a list of 10 isn’t all that hard.

When I got to the Hunger Games series, things got a bit more interesting. Starting with Catching Fire the bluray shows you 100’s of feature length titles – all the same duration, but with different segment maps. It seems that all but one of these titles has things shuffled in the wrong order.

Initially I naively picked the 1st and used that one, but upon watching the movie it was obvious there was 1 scene out of place, and thus two weird jump cuts to the flow of the movie. Enough to be annoying.

If you got digging around, you can find advice on which of the many to pick from the list. It turns out that there are multiple versions of the movie: rental, US release, Canadian release, etc. It’s much better to figure it out for the disk you have. The MakeMKV forum has a post about using PowerDVD and Process monitor to figure this out. I struck out here as I didn’t have a Windows machine with the right software combination, and it seems I wasn’t able to get Ubuntu to natively play back a bluray either.

Recently I came across a way to use MakeMKV to do the full process, again thanks to a MakeMKV forum post.

    1. Use MakeMKV to back up the full disk.
    2. Use MakeMKVcon to dump info to a text file

    3. Isolate the segment lists from /tmp/xx.txt. It turns out that “,26,” is unique enough to grab all of the segment lists. For Mockingjay Part 1 there are 550 segment lists on the disk, 519 of these are the length of the movie.

    4. Observations:
      a) All the movie length lists start with the same segment: 519
      b) It seems they all end with 520
      c) There are only 20 chunks in each segment list, and we already know 2 of them. Only 18 to sort into order correctly
      d) The MakeMVK backup has all of the chunks in backup/<disk>/BDMV/STREAM/
    5. Now we just need to play a copy of the movie, I have the DVD as well so VLC can play that back for me. Start at the first chunk, verify it is the start, watch the end to determine the scene break. Then figure out what the next chunk is.
      By building an incrementally specific grep, I can figure out the next chunk options. It is fewer choices than you might imagine. Each one had 2-4 possibilities.
      Hint: as you identify chunks, record the duration – this helps figure out where on the DVD playback you need to review to find the scene break.
    6. Once we identify the correct chunk order – we can go back to MakeMKV and rip the correct stream. A web search can also help verify which one is the right one, as I did for Mockingjay Part 1.

It took me about 35mins to get through step 5, much shorter than watching the whole movie. During the course of the chunk identification, I came across 3 where I had no choice, the only next chunk was the same one. After walking through 12 chunks, I hit a point where there was only 1 segment list left. I quickly verified the segment end/start matches and then double checked against the web search.

Docker and macvlan networking (IPv4)

Docker is the well known spin on Linux containers (LXC), if you’re not already playing with containers it’s probably time to jump in and get familiar. I’ve been (very slowly) migrating my personal infrastructure over to a container centric setup.

For me, containers are really nice for managing the set of software dependencies needed to run any particular application. It allows me to keep my RSS feed reader up to date, and avoids me breaking something my WordPress install needs or vice versa. Containers are a light weight virtualization.

The default networking model (default bridge) allows you to easily expose (map) a set of ports from the container, onto the host. This makes it easy to host an nginx container as your webserver on port 80.

Docker does some interesting network tricks to keep things more secure, but this gets problematic too. Containers can’t easily see the host they are on, making it difficult for container A to see container B’s port on the host – however, you can put both containers on the same docker network to allow them to see each other. This is a subject for another blog post entirely.

The macvlan support in docker is very cool. It allows you to provision a second IP address on the same network card, giving your docker container a full IP on the local network. In the world of virtual machines, similar macvlan support is available, and when you want to treat a docker container like a mini-VM, this is very useful.

Continue reading “Docker and macvlan networking (IPv4)”

Switching from DSL to Cable

I’ve been on DSL forever. I started out on Bell, have been on NCF, and most recently TekSavvy. I’ve had my trials and tribulations with DSL, and have a collection of DSL modems (some are backup, some are bad, some were sensitive to line conditions).

Cable has always been a faster alternative, but it meant I needed to pay a cable install fee and switch technology in general. Also, a static IP wasn’t possible on cable – and having self hosted lowtek.ca for a long time, I’ve always felt a bit trapped to DSL to give me a static IP.

DSL can be fast, but not in my area which seems to have been left behind for faster connectivity. The highest speed I could get was 15 down, 1 up. Now, 15 down is great – I can stream HD Netflix without any real problems. The internet feels fast enough.

I have to say, I really appreciate that Google has built a good enough speed test that is easy to use.

One motivating factor was my desire to stop paying Bell for my land line. It’s nearly $50 a month and we barely use it. Sure I could switch to a VOIP provider but then I still have to pay for the dryloop cost and it sounded like I’d probably experience a service outage (of up to 5 days) when the line switched.

Moving to cable means losing the static IP. It also means that outgoing port 25 is going to be blocked and this means my self hosted email server will have problems.

I’ve sketched out a solution for static IP hosting. I’ll try to write that up in the future once I’ve done it. For now, because on cable your IP rarely changes – I’m just pretending that the IP I have is a static IP.

For sending email, I needed to route all my mail via Teksavvy, treating them as a smarthost. My email setup is postfix, thus there is literally a one line setup change to /etc/postfix/main.cf

This works because TekSavvy allows anonymous SMTP to this from inside of the network they control. What it does is force all outgoing email to be sent to their server, which in turn will send it out.

Now modern email servers do additional trust checks, one of these is Sender Policy Framework (SPF). Configuring your SPF is done via a TXT record in your DNS. While the relayhost was working, I was seeing a warning when checking email sent to gmail (but other email providers check SPF too).

It took a while for me to figure out how to get my SPF record setup correctly. I got a bit lucky, as I was readingĀ https://support.google.com/a/answer/33786?hl=en which pointed at _spf.google.com as the Google SPF record holder. It turns out TekSavvy adopted the same naming: _spf.teksavvy.com. Your SPF record needs to point at other SPF records, so finding this meant it was an easy change to my DNS TXT record for SPF.

I will point at MXToolBox as a great web based tool for sorting out all sorts of email issues.

Now email was not only working via the smarthost, but my SPF record was setup correctly. I’m still experiencing delays when sending to gmail, but not apparently to other sites. From looking at the headers, it seems TekSavvy can at times (often) delay delivery to gmail. This is frustrating, but there are other paths to solution if it’s a big problem.

Now that email was sorted out, switching to cable was really easy. The cable box arrived and was “installed” by finding the live cable in my basement by the power panel and plugging the box in. It turns out that since the cable comes from a box at the end of my lawn and the buried cable to my house (which is 20yrs or more old) is in good shape, I get fantastic signal strength. The tech had to install an attenuator to reduce the signal to the happy range where the modem was going to work well.

Switching from DSL with static IP to Cable with rarely changing IP was a simple matter of swapping the WAN cable into my router from one box to the other. I had to reconfigure my router to use “Automatic” from “PPPoE”, and boom I was on the internet again. Visiting https://www.whatismyip.com/ and I had the new IP address, followed by a simple DNS change to use that as the address for lowtek.ca and I’m back.

At this point all I’ve lost is the reverse DNS check is failing, because the IP that lowtek.ca resolves to – does not answer lowtek.ca when you look up that IP. This is more important for sending email than receiving, and since I’m sending via Teksavvy it doesn’t matter as much. I still want a more ‘proper’ static IP to be assigned to lowtek.ca – more on that in a future post.

Boom – cable is just faster than DSL. With the added bonus that changing speeds is zero admin costs. If I want to move to 2x faster, it’s another $7 a month. On DSL I was at the fastest speed available to me at my location. Cable is costing me about $14 more a month, but the phone line savings will make up for that – once I get past the hump of buying a new cable modem and VOIP ATA box.