Mesemb Study Group
SPECIAL OFFER UNTIL 31st JANUARY 2008
Join now for 2008 and get the October 2007 Mesemb Study Group Bulletin free
UK £.8.00, Europe and Rest of World surface £10.00, Rest of World Airmail £15.00
Subscribe for 2007 and 2008:
UK £15.00, Europe and Rest of World surface £18.00, Rest of World Airmail £25.00
I have just sent this email to all our customers. I am aware that the list of email addresses I am using may have some obsolete addresses so if you do not get this sometime this morning please let me know at what address you would like to receive such messages:
I would like to thank all our customers for extreme patience over the last very difficult week as well as those of you who have sent messages of support. Suzanne and I have had a particularly difficult four weeks - some of you know some of the details and more will come out over the next few.
The server seems a little more stable although I have no particular confidence in there not being more breaks of service. You can contact me on firstname.lastname@example.org if necessary during an outage.
As you probably know I have been looking urgently for a new server to move all our domains to. I had been told over the last year that our Alabanza server was looking increasingly expensive. Having now got a feel for costs from other providers I can agree that the cost was above average although not hugely so - perhaps 15-20% - and in former circumstances that would not have made me move because of the good support I got from Alabanza staff and the amount of work involved in a move. A lot of the cost related to the large amount of data moved in and out of our sites (I was purchasing 700 Gb per month, although not all of it was used - peak month earlier this year was around 400 Gb)
One thing has emerged and that is I shall be able to get a lot more disk space for the same price. I will be allocating you all at least double for the same price as some compensation for the problems. Some sites will not perhaps need this but they can in that case allocate it to one of the Society sites who I am sure can find uses for it. I would hope that once past this initial financial and technical crisis (and more of that elsewhere) I will be able to double this allocation again by next August.
For some of the higher cost sites I also hope to be able to offer some reduction in price assuming there is not an exodus of customers.
One thing I will be strict about is the some half dozen hosting customers who are excessively behind with their bills (in some cases more the a year) I can no longer afford to carry these sites.
I have a number of technical issues still to resolve about certain specialised facilities and I will be contacting the small number of customers involved to discuss the best solutions. This only involves domain hosting customers. Advertisers and those with pages on cactus-mall.com will not be affected in any way.
I have had a number of comments on slowness of the server since the move. I have found it somewhat erratic - occasionally very slow but mostly unchanged from before the move. Some of it was undoubtedly routing tables and the like settling down, some due to catching up with backlog of mail but I think also there may be a large effect from customers from all the moved servers copying all their data from the systems to move to other suppliers. Which brings me to another point. Those of you responsible for your own domain should make sure you have
an up-to-date copy of your website on your local system. When we move you will need to reload your files. Those of you with Forums etc may need to copy MySQL databases as well. If you don't have your own expert in doing this probably one of our more experienced user can help.
Mesemb Study Group Bulletin
Mesemb Study Group Bulletin Vol 22 No 4 was posted today.
Observations in the field on Delianthe peersii (L.Bolus) N.E.Br. by J J du Toit
Lithoparian Recollections - Part 10 by Des & Naureen Cole
A ‘Conothon’: Conophytum Hunting in the Western and Northern Capes - Part 1 by Andy Young X Gibbaeophyllum `Grand Marnier' - A Mixed-Marriage Bonanza by Gordon Rowley
Navigation by Divagation by Steve Hammer
Conophytum Herrei, Schwantes by Dr. N.E. Brown
Further Divagations by Steve Hammer
Mesembryanthemum by T.M. Endean (Resumé of a talk given on April 24th, 1935)
The Genus Mossia, N.E.Br. by N.E. Brown
Lithops terricolor cv. Speckled Gold by Bernd Schlösser
Literature Reviews by Suzanne Mace
Downtime was about 30 minutes
Now back up
More Server problems
All websites on the server are currently giving connection refused messages.
I have opened an emergency fault message.
At this time mail, FTP and DNS seem to be still working.
Cactus-Mall Domain stats
It has been reported to me that the Stats package on our domains are not updating. I have issued a ticket through the help desk which is fortunately now working again.
Situation re compensation from Alabanza/Navisite
My contract with Alabanza includes a Service Level Agreement. I assume since Navisite took this contract over and indeed charged me under it they will be liable for the same payment.
If this is forthcoming I will of course feed back to customers who have incurred financial losses due to the outage. If not I will have to consider legal action against Navisite, but of course this is likely to be a very protracted procedure. I am not quite sure if non-USA residents can join the Class Actions which are proposed. I have already been approached by the ambulance chasers over this.
Use of the Blog
This Blog has been very useful over the crisis of the last few days and has convinced me of the value of a Blog over which I was a bit skeptical in the past. I will try and keep up with a bit more regular entries in the future and will hopefully attract a more regular audience.
I have sent out a general message to customers over the problems of the last few days but I am fully aware that in the current email climate not all of these get through. If you think you should have received this email try adding email@example.com to your friends list or perhaps send me an alternative email address which may be less heavily filtered.
To give you a few more items of news. This Thursday will see the publication of the last edition of the Mesemb Study Group Bulletin for 2007. The good news is that membership of the MSG is actually up on 2006 (any other groups match this?). Membership costs for 2008 are unchanged except that we have had to increase the airmail subscription outside Europe due to increased postal costs by £1.
Last sunday the Convention committee of the BCSS had its first meeting to consider the next BCSS Convention. Suzanne was elected Chairman of this committee which has a number of new members and lots of ideas were floated to make this a different event from previous conventions which will appeal to a wider spectrum of members. Several venues will be inspected in the near future and prelimanary proposals made to the F&GP of the BCSS. We will then be able to suggest a date and venue for your diaries.
Note: Blogs to this site recently have all apparently come from Suzanne, whereas actually they might have been from either myself (Tony) or Suzanne. This was due to limitations imposed by the absence of the server which had also sorts of complications on our internet access. I will get this sorted in the next few days so that it is clearer who is posting.
Back on-line now
Back for the moment - but god knows for how long.
I will endeavour to answer some of your emails before it goes again.
It has all gone again
If I was in Baltimore/Andover now I would be strangling somebody.
It was mostly working for about two hours, Now nothing is working again, no mail, no FTP, no HTTP, no DNS resolution. This is totally impossible.
It seems most things came up somewhen between 2.00am and 7.00am on the 6th November when I was in bed. I suspect it was toward the later end of that period, so well over 60 hours downtime. If anybody can exactly pinpoint the time it would be useful in my claims for compensation. (I cannot use data from other Albanza hosts as the time they came back varied by at least 12 hours and some are still not up). I have only found one or two problems so far and have been able to fix them. Reports of any others would be useful.
I will not repeat the statements which I have posted to the Cactus-Mall and to the BCSS Forum, but it would be most helpful if the situation can be relayed to other news sources, forums etc - particularly in non-English languages so that all are informed as to what has been going on.
It would also be useful if any of our commercial customers can quantify likely costs to their businesses so that I can guage what is needed in terms of compensation.
Not much more progress
I regret I still cannot contact any of our websites by HTTP. They either give 'connection refused' and time out. Those that give connection refused I can contact by FTP. email I think is working on those that I can contact by FTP but not on the others.
I have successfully got through to our account manager and requested that the apache http module is restarted. He has sent this to the engineers but if they have done it there is no result yet.
Cactus-Mall FTP success
It is possible things are moving - I have just been able to access www.cactus-mall.com by FTP and load a new message to the front page.
I have not yet managed to load it in a web browser, but looking on Google Urchin there are records of a few accesses so I gues a few may have suceeded
Another source of information
It does seem from this link that some ex Alabanza sites are coming back on-line, although I have yet to have an luck with any of ours. I can get some traceroutes to work however.
Latest Navisite messge
Latest message from Navisite suggests some Alabanza sites are up although I have yet to actually find any that are,
They say the remainder are affected by a 'DNS issue'. I find this hard to beleive since none of the sites are contactable by IP address. I suspect the real problem maybe that they have screwed up the routing tables.
Cactus-Mall server downtime.
We are now up to 50 hours downtime on all our websites and I have had no update from Navisite since 3.00am GMT this morning. No answers to any emails.
Some have suggested posting tickets to the Alabanza help desk. Very thoughtfully Navisite have included this system in the move, perhaps this was intentional.
I have been combing the web for information on this bunch of cowboys and note that Hewlett-Packard are one of the major shareholders so I have accordingly alerted them to the situation.
Data Center Move Fiasco
The above link contains a fair summary of the current position.
Data Center Move
This link may illuminate the current position:Alabanza and Navisite
Data Center move
Here is the latest set of
messages from Navisite. Pity they don't say which day (or year) it refers to.
Message at 16.18 GMT
We would like to update you on the status of the Alabanza Platform migration. At approximately 7:30 AM EST, the netblocks were brought online and routing traffic to the new platform in Andover. Shortly thereafter, significant network instability was encountered and the netblocks were removed.
The design of the platform is such that significant LAN traffic is generated. As you know, there is a portion of the platform that is being shipped from Baltimore to Andover, and a review of the network traffic is showing an extreme amount of packets destined for servers that are not on the network.
A team of Network Engineers is provisioning an interim solution while the servers are in transit from Baltimore to Andover. The ETA of the servers still in transit is 5:00 PM EST and the interim solution is targeted for 12:00 PM EST.
Message at 17.59 GMT
We would like to update you on the status of the Alabanza Platform migration. As stated in the last status update we continue to experience a high level of traffic saturation and network instability.
Our Network Engineers are continuing to build out a solution and we anticipate that they will have that in place no later than mid to late afternoon.
Help with getting information out
If somebody could relay the information about the the Cactus-Mall server sites downtime to cacti_etc and other mailing lists it would be most useful. Unfortunately I am subscribed to most of these using firstname.lastname@example.org and I cannot get mail out from this address even using alternative smtp servers as they insist on the address resolving.
Data Center Move
Nearly 9.00am in the UK, 23 hours since the server was taken down and still no further news. I have posted many compliants throgh the few chanels that are open to me (most are totally clogged) but had no replies.
I am sorry that all Cactus-Mall sites are still down but this is totally outside my control.
Here is the latest statement from Navsite but I have pretty low confidence in it as they seem to be extremely incompetant. Please feel free to complain bitterly to them:
We would like to update you relative to the timing and status of the data center move. As of
this update, the Rsync process has encountered technical issues which has forced us to move
the migration window roughly to 6 AM EST. Please note that services will not be fully restored
until all hosts have been moved and the netblocks have been moved. We anticipate that we will
have these technical issues resolved shortly and have this process move forward through out
the night. We will continue to update you as the move proceeds. Thank you for your patience
as we focus on completing this move.
If you have any questions or concerns, please send us an email at email@example.com
or call us at the number listed below.
Client Care Group
978-946-5999 (Outside the US)firstname.lastname@example.org