There was a foul-up with the vis.nu domain today, so it was unavailable for most of it. Mail, unfortunately, will have bounced. It may take up to a day or so for new DNS changes to propagate, but it was solved this morning.
We’re doing a major database upgrade today. The expected upgrade path is to dump all databases, upgrade, and then reimport. Since this is essentially completely rebuilding the DB structure, I’m taking this time to bring the db server more in line with how we do things. Actual downtime should be measured in minutes, but changes may be lost if they are committed today.
I’ve disable the SSD flashcache plugin, for a couple of reasons explained after the break.
More importantly, I’ve decided to shut down Indra, the virtual machine what provides shell access to people logging into vis.nu. The reasons are very simple– no one uses it. vis.nu has evolved since 1998, and where almost everything was done via shell back in the day, these days most people use web interfaces. If they need shell, they use their local shells, for the most part. Even I use a VM on my local network rather than log into indra. So to save resources, and maintenance time on my part, I’m shutting it down.
Should we need to bring it back, it’d be relatively trivial. SSH services (the only thing really handled by indra) will now be directed to kali, the web server.
Looks like whatever made Apple block my domain is gone. Hooray for that.
I’ve also updated webmail and associated interfaces. You can check it out here. Changes:
- Upgraded to the latest software
- HTTPS is now enforced
- Mail client has HTML5 notifications
- keyboard shortcuts
- drag and drop upload
- you can now change your mail password and forwards
- and a bit more…
I’ve submitted a question to the blocking authority, but no answers so far. Please let people know. I’ll update again when I know what the hell is going on.
New Mail SSL Certificate
I have purchased an SSL certificate for mail.tacomatelematics.com. So, in your mailer, as long as you use “mail.tacomatelematics.com” as your mail server, it will no longer complain about a self-signed certificate.
This also means that SSL is now fully functional for the webmail client. Since we no longer have the self-signed certificate concern, I will be disabling webmail access via standard HTTP. Finally.
Why hadn’t I heard of SNI years ago?
Web Server Update
The hard part of upgrading the server is done now. So that’s good. Aggregate downtime was about five minutes, hooray for virtualization.
I’m going to be adding some other things to the web server (in anticipation of wrapping the media streaming and distribution functions out of the media server), but that will have downtimes measured in milliseconds.
I’m upgrading the web server today. In fact, the web server we’re using right now is the backup image of the actual web server.
There should be a couple of outage about a minute or two long, but web performance is going to be slow as I make changes in LVM snapshots, and make backup copies.
This should all be done by tomorrow, the 11th of June.
Database server has been updated, which is why the web server was showing a maintenance page for a little under an hour tonight. No data lost, mail queued, everything went fairly painlessly.
Next is the mail server.
Copernicium got strange. Copernicium, or Cn, is the server in Arizona that all the main VMs run on.
I can’t say it crashed, because I rebooted it. It just got hung up on a bad XFS filesystem and wouldn’t properly remount. I needed to have someone at the facility go up and press the button.
Sort of my bad, but I was attempting to begin upgrading to fix some problems on the network, so I need to work on this. Total outage about eight minutes.
UPDATE: Now that that’s out of the way, I’m beginning site-wide upgrades. I’ve done a test upgrade of of indra and it is working, so I’ll start with that one.
Okay, the server move is done, and we’re even on the same IP we were on before we moved. The machines will still answer on the old IPs for the next couple days, but it looks like world DNS has caught up and I’m only seeing a few bytes a second over the old link.
Also, I’ll take this move as an opportunity to close out some old stuff that hasn’t been updated in a million years, over the next few weeks. Old domains, projects, email accounts, things like that. I’m not deleting anything or removing anything that looks like it’s been used recently, but if I do, email me and I’ll fix it as soon as I can.