We have, for some time, been working on the next generation of LNS (L2TP termination) code in FireBrick, and that is ready for live testing (initially just test lines and staff lines)...
We are planning to deploy this over the weekend. This should not disrupt customer traffic as it is being set up as a separate box on the network. However, with BT having slightly flaky routing at present we cannot be 100% sure.
There are only minor changes that this new software offers end users including a slight change in the graph layout and visibility of BRAS changes during the day on the graphs. However, it is the platform for the further development of the priority traffic tagging and other features moving on. It also has much better command line diagnostics we can use when trying to debug BT issues such as this mornings.
Friday, June 12, 2009
Saturday, June 06, 2009
21CN work this weekend
Following the issues yesterday I am keen to get to the root cause of this. We believe it is a specific BT router. I will be trying to get BT to work on this over the weekend rather than waiting until Monday. This means we may have to work with them to confirm things are working correctly, change routing, etc. This should not break things, but may mean ppp restarts with a few seconds outage if we have to move sessions about. I'll try and avoid this, but I'd rather get this sorted during the weekend than during the weekday. Please bear with us.
Thursday, June 04, 2009
Sunday maintenance window
We are working on updates to router code and expect to be trying out some new code on the beta test routers over the next few weeks. However, we have some minor improvements that we can load in the mean time.
This means we may be loading new code on live routers, so the plan is to have a maintenance window of "early Sunday morning". Lets say up to 15 mins some time between 00:00 and 07:00, which in practice means either someone stays up late Saturday and does a reload just after 00:00, or someone gets up early and loads at 06:30, or just possibly both on rare occasions.
If we do this, it may mean a few seconds off line for a PPP restart and could be 20CN or 21CN lines affected depending on the router. I'll mention whats happening on irc at the time.
One of the reasons to do this, by the way, is to try and speed up recovery. In the past we have seen BT take 20 minutes to reconnect to us, and have seen load on RADIUS servers take minutes to connect everyone. We think we already have a number of improvements that should reduce things to a matter of seconds or at most a few minutes, and at the very least we need to understand what BT do for 20CN and 21CN and IPSC connections properly.
Obviously as we get bigger it is important that our infrastructure can recover as fast as possible in case something happens. If we find BT lacking in these type of tests it is useful to have evidence to take to them to get them to improve matters too.
Thank you for your understanding.
This means we may be loading new code on live routers, so the plan is to have a maintenance window of "early Sunday morning". Lets say up to 15 mins some time between 00:00 and 07:00, which in practice means either someone stays up late Saturday and does a reload just after 00:00, or someone gets up early and loads at 06:30, or just possibly both on rare occasions.
If we do this, it may mean a few seconds off line for a PPP restart and could be 20CN or 21CN lines affected depending on the router. I'll mention whats happening on irc at the time.
One of the reasons to do this, by the way, is to try and speed up recovery. In the past we have seen BT take 20 minutes to reconnect to us, and have seen load on RADIUS servers take minutes to connect everyone. We think we already have a number of improvements that should reduce things to a matter of seconds or at most a few minutes, and at the very least we need to understand what BT do for 20CN and 21CN and IPSC connections properly.
Obviously as we get bigger it is important that our infrastructure can recover as fast as possible in case something happens. If we find BT lacking in these type of tests it is useful to have evidence to take to them to get them to improve matters too.
Thank you for your understanding.
IPStream connect
We are working with BT on the IPStream connect trial at present (20 customers) and this is basically "just working"! We have a few more weeks of tests to confirm things are all fine, but it is going pretty well.
So we may be able to go ahead with the full scale move to IPStream connect in one or two months. We don't have a date yet or the full commercial details from BT on this just yet. Obviously we are asking the questions you expect including "how do we reverse out if things go wrong?".
When this happens it will basically mean lines moving end point, which we can start gradually over a day or so and then move everyone else over night. It should be pretty seamless and be a matter of seconds to change over. There is no change to configuration or IP addressing at your end.
From a customer point of view this will have very little impact. 20CN lines are already managing well within the capacity of the BT Central links now. It does remove some of the commercial reasons for stopping the 21CN migrations, so we are likely to resume those as soon as we are happy with BT on a technical front.
From our point of view it makes various commercial aspects of the way we work with BT more flexible and starts to save us some money (which is why we recently reduced prices). Sadly it is not as much as we had expected, but every little helps. It also does mean we have no major restrictions on customer volumes and growth, and will make it easier to offer some more wholesale services to small ISPs and corporate customers.
We'll let you know when we have a confirmed date.
So we may be able to go ahead with the full scale move to IPStream connect in one or two months. We don't have a date yet or the full commercial details from BT on this just yet. Obviously we are asking the questions you expect including "how do we reverse out if things go wrong?".
When this happens it will basically mean lines moving end point, which we can start gradually over a day or so and then move everyone else over night. It should be pretty seamless and be a matter of seconds to change over. There is no change to configuration or IP addressing at your end.
From a customer point of view this will have very little impact. 20CN lines are already managing well within the capacity of the BT Central links now. It does remove some of the commercial reasons for stopping the 21CN migrations, so we are likely to resume those as soon as we are happy with BT on a technical front.
From our point of view it makes various commercial aspects of the way we work with BT more flexible and starts to save us some money (which is why we recently reduced prices). Sadly it is not as much as we had expected, but every little helps. It also does mean we have no major restrictions on customer volumes and growth, and will make it easier to offer some more wholesale services to small ISPs and corporate customers.
We'll let you know when we have a confirmed date.
Tuesday, May 12, 2009
[Closed] Telecity Hex Planned Rack Closure 31st May
We will be closing down a rack in Telecity Hex 6/7 on the 31st of May, Customers will be notified of this move, an engineers number will be given on the day for any issues customers may have.
The Move will only be from 1 rack to another and downtime will be kept to a minimum.
When instigating a similar move on a previous occasion servers were down for an estimated 10 minutes.
If customers would rather move to our datacentre in Maidenhead this can be arranged but downtime would be around 4 hours.
please contact support for any questions.
The Move will only be from 1 rack to another and downtime will be kept to a minimum.
When instigating a similar move on a previous occasion servers were down for an estimated 10 minutes.
If customers would rather move to our datacentre in Maidenhead this can be arranged but downtime would be around 4 hours.
please contact support for any questions.
Subscribe to:
Posts (Atom)
Posts
