We’re doing a major database upgrade today. The expected upgrade path is to dump all databases, upgrade, and then reimport. Since this is essentially completely rebuilding the DB structure, I’m taking this time to bring the db server more in line with how we do things. Actual downtime should be measured in minutes, but changes may be lost if they are committed today.
I’ve disable the SSD flashcache plugin, for a couple of reasons explained after the break.
More importantly, I’ve decided to shut down Indra, the virtual machine what provides shell access to people logging into vis.nu. The reasons are very simple– no one uses it. vis.nu has evolved since 1998, and where almost everything was done via shell back in the day, these days most people use web interfaces. If they need shell, they use their local shells, for the most part. Even I use a VM on my local network rather than log into indra. So to save resources, and maintenance time on my part, I’m shutting it down.
Should we need to bring it back, it’d be relatively trivial. SSH services (the only thing really handled by indra) will now be directed to kali, the web server.
Looks like whatever made Apple block my domain is gone. Hooray for that.
I’ve also updated webmail and associated interfaces. You can check it out here. Changes:
- Upgraded to the latest software
- HTTPS is now enforced
- Mail client has HTML5 notifications
- keyboard shortcuts
- drag and drop upload
- you can now change your mail password and forwards
- and a bit more…
I’ve submitted a question to the blocking authority, but no answers so far. Please let people know. I’ll update again when I know what the hell is going on.
New Mail SSL Certificate
I have purchased an SSL certificate for mail.tacomatelematics.com. So, in your mailer, as long as you use “mail.tacomatelematics.com” as your mail server, it will no longer complain about a self-signed certificate.
This also means that SSL is now fully functional for the webmail client. Since we no longer have the self-signed certificate concern, I will be disabling webmail access via standard HTTP. Finally.
Why hadn’t I heard of SNI years ago?
Web Server Update
The hard part of upgrading the server is done now. So that’s good. Aggregate downtime was about five minutes, hooray for virtualization.
I’m going to be adding some other things to the web server (in anticipation of wrapping the media streaming and distribution functions out of the media server), but that will have downtimes measured in milliseconds.
I’m upgrading the web server today. In fact, the web server we’re using right now is the backup image of the actual web server.
There should be a couple of outage about a minute or two long, but web performance is going to be slow as I make changes in LVM snapshots, and make backup copies.
This should all be done by tomorrow, the 11th of June.
Database server has been updated, which is why the web server was showing a maintenance page for a little under an hour tonight. No data lost, mail queued, everything went fairly painlessly.
Next is the mail server.
Copernicium got strange. Copernicium, or Cn, is the server in Arizona that all the main VMs run on.
I can’t say it crashed, because I rebooted it. It just got hung up on a bad XFS filesystem and wouldn’t properly remount. I needed to have someone at the facility go up and press the button.
Sort of my bad, but I was attempting to begin upgrading to fix some problems on the network, so I need to work on this. Total outage about eight minutes.
UPDATE: Now that that’s out of the way, I’m beginning site-wide upgrades. I’ve done a test upgrade of of indra and it is working, so I’ll start with that one.
Okay, the server move is done, and we’re even on the same IP we were on before we moved. The machines will still answer on the old IPs for the next couple days, but it looks like world DNS has caught up and I’m only seeing a few bytes a second over the old link.
Also, I’ll take this move as an opportunity to close out some old stuff that hasn’t been updated in a million years, over the next few weeks. Old domains, projects, email accounts, things like that. I’m not deleting anything or removing anything that looks like it’s been used recently, but if I do, email me and I’ll fix it as soon as I can.
As most of you know, I’m closing my office in Tacoma and combining my work and living situations. Part of that move is relocating my main servers out of a rack that I keep to a recently built server at a colocation facility in Phoenix, AZ. I’m happy to report that the server is built, shipped, and booted at the new facility, and all of the virtual machines that were running in Phoenix already (mostly podcast distribution) have been running on the new hardware for several days.
Today, I’m moving the virtual servers in Tacoma to their new home. This means shutting down each server one by one, refreshing the copy on the new server so all the data is up to date, and starting it on the new server. The process takes about half an hour for each server, more for some, less for others. During the copy time, those services will be unavailable.
Once this is done, I’ll be changing DNS entries to point to the new IP address, or contacting people to make that change if I cannot.
I’ve set up a dual-routing and VPN system that’ll let the new servers respond at the old and new IPs at the same time, cutting out transition outage time. It does mean that access at the OLD IP will be a bit slower and have higher latency, since all requests go back out over the network to Phoenix. But it will work as normal until we’re fully moved out of the old IPs.
If you have any questions, please contact me.