I want to look into how we do DNS these days. There are a couple of choices – one of the reasons there are choices is that DNS kind of sucks. But it also drives the Internet. I am old enough that I remember, having just missed the era of programming mainframes with punch cards, that before DNS all we had were 4-octet IP addresses. Basically the 8.8.8.8 or the 127.0.0.1 or whatever you’ve seen and perhaps wondered about.
Here’s where we are in the table of contents (bolded):
- Decide on regular DNS or Dynamic DNS to provide an external DNS hostname so other peers can find my instance.
- regular DNS
- Domain: perivid.xyz
- Registered, registrar chosen, domain chosen.
- DNS A record pointed at my external WAN address (that rarely changes)
- Domain: perivid.xyz
- Keep an eye on my external address for any changes (there are utilities you can use to watch the external address and notify you when it changes – also my external IP address has only changed once in 3 years, so I’m not super concerned)
- regular DNS
- Configure router/firewall to activate separate network on second LAN port, and configure firewall.
- Acquire hardware:
- (These steps per YouTube video on how to set up PeerTube on a Raspberry PI 4)
Install Raspbian LITE on 256 GB microSD card - Attach external drive to Raspberry PI and partition it, and verify it works as a filesystem.
- Install Dependencies for PeerTube
- Install PeerTube
- Configure PeerTube
- Test PeerTube
- Troubleshooting
- Set up port forwarding rules on router to forward traffic to the Raspberry PI instance of PeerTube
- Troubleshooting
- Move forward with creating user account and configuring both the Instance and my user account Channel, playlists,, etc.
- Figure out how to federate instance with other instances.
I’ll update this outline/table of contents as more steps becoming apparent via the how-to and related videos.
Weird that we call these 1 to 3 digit numbers octets, right? Well the reason we call them octets is that they are represented in computer land as 8 bit numbers, so e.g. 1 = 0bx00000001 and 127 = 0bx01111111. Don’t worry about the 0bx part, that’s just how we designate that this number is a binary number, not a decimal one. Anyway, note that there are 8 binary digits after 0bx, which is how we can represent the decimal values in 4-octet IP addresses, which is why, then, they’re called octets! Cool huh?
Anyway, the days before the DNS-enabled Internet basically meant that you had to look up and remember, or maybe watch other users use 4-octet IP addresses, and later poke at them to find out what they were and then use them yourself. And around the time the Internet went truly commercial with shops and big corporations, that’s sort of when DNS started going into widespread use. Now you pretty much cannot go into the Internet with just an IP address, except maybe in Minecraft and other games that allow small time personal game servers. (And as a brief digression, thinking about it, I just assigned a resolvable subdomain to my Minecraft server from where I’m hosting the domain I picked for the PeerTube.)
So now for my project I need a DNS subdomain entry for the PeerTube host. It’s going to live on the my local network in its own DMZ network behind the external IP address my whole house has. Normally that doesn’t matter, because we’re just browsing and using Internet services. But to run a server, my Firewall has to take incoming traffic and forward traffic on the right ports to the Peertube instance so it can answer web browser requests. And for that, I need to set up a DNS pointing to my external address, which is technically dynamic, or at least I am not guaranteed a static, guaranteed external address, nor can I even upgrade to buy one. They’re simply not available on my service.
And you may say “change services!” but for now they’re the only gigabit fiber option I have that allows me to host my own server/services (noncommercially), so I’m keeping it, despite the need to write this custom dynamic dns script/tool and deploy it somehow.
These days, the answer to this kind of challenge, to change DNS to IP mapping on the fly (if my Internet provider decides to change it), is to either use a Dynamic DNS Service (DynDNS) enabled router or to use a normal DNS host and a script to adjust the IP address for the DNS when the external address changes. Since I’m already using a normal DNS service, I should probably use a script.
Now the polite scripts check for a change and only then do they update the DNS records. A less polite one would run every X minutes, hours, etc., and just update the normal DNS Services. This is tempting but it’s a good way to get blocked by your hosting service, so I think I would like a polite version.
The bad news is that since most of the world relies on Dynamic DNS to do this kind of thing, there don’t seem to be a lot of pre-written off-the-shelf applications that do exactly what I need. (Most will either only update known services that lots of people use, or they’ll email me or text me or some other manual way of notifying me – but then I’d have to manually go change the DNS settings, which is what I’m trying to avoid.) But the good news is that I can write a script or some utility that can do this for me, and further, my web host’s DNS service has an authenticated web-based API that I can use to update my DNS records.
So basically I just need a tool that can periodically look up my external address, like icanhazip.com, recently acquired by CloudFlare), track it for any changes, and if it changes, update my DNS records via a specially-formed URL. Then I can run it every hour (as a cron job or scheduled job) and it should work in perpetuity. (Note: it doesn’t look like CloudFlare will ever get rid of icanhazip but I suppose it’s possible, so I should keep an eye out.)
Anyhow, finding a custom dynamic DNS application that I can use for my own devices was a fruitless search for me. It’s not like this should be all that hard, but even though it’s a small and particularly unambitious project, it still has huge PITA potential, so I can see why some folks shell out tens of dollars on a solution to this that is someone else’s.
Nevertheless, since I turned out to be totally uninspired by the commercial, off the shelf packages I found (that didn’t seem to have the flexibility I needed), I decided to roll my own. Figuring that this need would persist as long as I had the router, I decided to put it on the UNIX/Linux operating system that lives on my router, and register it as a cron job, so it was time to dust off the bash scripting hat and Linux admin hat and make my mark.
Note that there are many different layers to your network that you could choose to implement a utility like this on. If you have an always-on computer, you could run the tool/script there. If your router has a good operating system, you could run it there. If you have a self-hosted server, or an unused or underutilized raspberry pi, you could do it there. Each choice limits you to different technological approaches, including the basic architecture, the language if it’s a script or a full fledged language, and whatever you’re using to kick off timed jobs every day or hour or whatever you anticipate your needs would be. By my reasoning, the router is likely always (or at least for several years) going to be part of my tech stack, and is going to be handling other roles, like hosting the local network and DMZ where the PeerTube instance will live on the Raspberry PI, so it makes sense for the script that updates the DNS dynamically if my external IP changes to live there too. But you may have a different situation and therefore a different solution.
I found a close-enough blog post about a similar situation that Michael Ryom wrote up 4 years ago with the same router and a different DNS host with different scripting requirements. The main difference is that his DNS host allows updates, whereas mine requires removing the old entry and adding a new one, and further, when you remove the old entry, you have to know what its old IP address value was. So I had to figure out how to do a cron job script that could save data between executions, which I found in a different web writeup elsewhere. (Note: in the StackOverflow discussion there is a huge discussion about the line “value=$((value + 1))” which we don’t care about here since we’re not using it, just the bits to store and retrieve the value around that expression.)
Why yes, shell scripting can be a glamorous duty where you write everything yourself from novel routes and never go to the Internet, or, if you are working on a larger problem, you can go find examples you understand and crib liberally from them. The thing that matters is that you understand what you’re doing (so you don’t introduce glaring security issues) and that you move forward, and you are the missing piece, who takes two or three different solutions for related problems and rolls them together to make something beautiful. Or at least functional. Cribbing is fine. Just be transparent about it and link and credit and if people don’t like it, they can try doing it themselves. In a great majority of situations, they’ll prefer to pay or credit you for doing the syncretic work. You can find glamor or glory elsewhere, and if you do, you can celebrate your victories all you want.
Anyhow, I combined, tested different parts of the script (locally on a MacOS machine, since this stuff is basic and works anywhere not Windows, essentially), and came up with something that works pretty well. I’m hosting the version where I don’t provide my private API key on Gitlab. Notes if you decide to use it yourself:
- I do not include a working API key. You have to get one for yourself, if you are working with Dreamost’s DNS API (or refactor if you’ve got another DNS host, of course)
- Cron jobs execute as root, the super user, so be prepared for the saved value file, ‘value.dat’ to most likely end up in root’s user directory. On my router, the file lives in /root/value.dat.
- Speaking of execution context, if you test this script as root, it’ll create that ‘value.dat’ in whatever directory you’re in when you test it. So if you’re in /config/scripts/post-config.d when you test it, you’ll find an extra value.dat there. Just be warned. It shouldn’t really create any issues.
In case you’re curious, test1.sh tested the variable storage. test2.sh tested the log output and just the syntax of and parsing of the strings with variable substitution that would eventually make up the curl-sent commands. test3.sh went through iterations of final testing, figuring out things like execution perms and execution context and so on. Also the major reason test3 git.sh is so much bigger than Michael Ryom’s is that I tend to be more wordy in my comments, and I try to give links and references I’ll need in the future to figure out wtf I did when I forget.
Also I wasn’t able to fully test the script. I didn’t quite test the actual function of the cron script to the extent of updating DNS. I tested and proofread the curl-sent commands it intends to send when the external IP address changes, and I’ve manually sent those commands to get the result I need, but I haven’t made the script send the commands yet in the cron job where it lives. So if my IP does change (it hasn’t yet in several years) I’ll need to probably make sure it worked as intended.
Anyhow, here’s the full extent of what I did with the script, in my USG 3P hardware, most of this parroting what Michael Ryom did.
I found the Dreamhost DNS API documentation. I got a DNS API Authentication key. I squirreled it away. I’ll use “1A1A1A1A1A1A1A1A” in the rest of this but that’s not my real API key.
The primary commands are (using the fake API key, but my entry information, for the most part – replace also the 0.0.0.0 IP address with appropriate real value):
Remove DNS entry:
https://api.dreamhost.com/?key=1A1A1A1A1A1A1A1A&cmd=dns-remove_record&record=video.perivid.xyz&type=A&value=0.0.0.0
Add DNS entry:
https://api.dreamhost.com/?key=1A1A1A1A1A1A1A1A&cmd=dns-add_record&record=video.perivid.xyz&type=A&value=0.0.0.0
I tested and wrote the script that lives on Gitlab for test3 git.sh.
I put that inside the script framework Michael Ryom provides and kept it ready in a local text file.
I logged into my USG 3P (I don’t have cloudkey):
ssh <username>@10.4.1.1
After negotiating the login (if it’s your first time connecting, you check and accept the key, and then for all other connections you provide the password), I got to the shell prompt, and escalated my permissions:
sudo -i
This starts a second shell with your permissions escalated to root (the super user).
Michael, here, puts the prepared script into a file in the /config/scripts/post-config.d folder. He names his create-ddns. I named mine create-custom-ddns.
cd /config/scripts/post-config.d
vi create-custom-ddns
Within vi, you type the ‘i‘ key to go into insert mode, and then you paste the contents of test3 git.sh in, inside Ryom’s code, or, for completeness, this:
#!/bin/bash
cat << 'EOF' > /etc/cron.hourly/ddns
#!/bin/bash
# For Custom Dynamic DNS behavior for Dreamhost
# DNS host and bash-capable router
# init
# change to YOUR API Key
# (https://help.dreamhost.com/hc/en-us/articles/4407354972692)
DNSAPIKey="1A1A1A1A1A1A1A1A"
DNSName="video.perivid.xyz"
DNSRecordType="A"
# get current external WAN IPv4 address
existingIP=$(curl -s -L "https://icanhazip.com/")
# if we don't have a ./value.dat file, start with current value
if [ ! -f "./value.dat" ] ; then
storedValue=$existingIP
# otherwise read the value from the file
else
storedValue=`cat ./value.dat`
fi
# If there's no change, don't update anything
# If there has been a change, remove old DNS entry and
# add new one.
# NOTE: Dreamhost doesn't allow updating existing records, so this
# is the only way.
if [ $storedValue != $existingIP ] ; then
printf "IP CHANGED since last check\n" | logger
printf "Command to remove DNS entry:\n" | logger
curl -s -L "https://api.dreamhost.com/?key=$DNSAPIKey&cmd=dns-remove_record&record=$DNSName&type=$DNSRecordType&value=$storedValue\n\n" | logger
printf "Command to add new DNS entry:\n" | logger
curl -s -L "https://api.dreamhost.com/?key=$DNSAPIKey&cmd=dns-add_record&record=$DNSName&type=$DNSRecordType&value=$existingIP\n\n" | loggeer
else
printf "No IP Change" | logger
fi
# and save it for next time
echo "${existingIP}" > ./value.dat
EOF
chmod +x /etc/cron.hourly/ddns
After creating this, save it and exit by using the escape key to get out of insert move, and either use “ZZ” to quick save and exit, or you can use the command “:wq” to do the same thing.
Whenever the USG is rebooted, this script is executed, creating or overwriting an hourly cron job.
Piping the change or no change events to logger logs them to /var/log/messages, so you can see something happening with the cron job.
Assuming you don’t want to reboot your USG multiple times to test the script, you can execute it yourself. First, set the file permissions to allow execution:
chmod +x /config/scripts/post-config.d/create-custom-ddns
Then run the cron-script-creating-script:
sh /config/scripts/post-config.d/create-custom-ddns
Then run the actual script that lives in the chron folders:
sh /etc/cron.hourly/ddns
To check for your script’s activity, you can do either:
tail -F /var/log/messages
If you use this, you’ll have to Ctrl-C to get out of the tail viewing mode.
Or you can use the non-interactive:
cat /var/log/messages | grep logger
Great! Remember to (and note to self) monitor this and if the external IP address does change, make sure the cron job worked as expected.
This post is number 3 in a series. As it progresses, I’ll update the table of contents below so you can find other parts.
(Photo by Pavel Danilyuk: https://www.pexels.com/photo/man-and-woman-working-at-the-office-7654120/)