netbilling down?
 

anyone else having issues? i usually get a call, nothing this time. cant process transactions or access the admin. their www works fine but thats about all. tried calling, just get sent to people's voicemails.

I am not able to access the admin, I believe transactions are going through through, my sales and rebills as reported by nats are incrementing as expected including several in the last 30 minutes (actually a little better than expected but not enough to suspect something was wrong)

I am simply not able to access the admin to run reports or process cancellation requests.

I called to talk to support and they said they were on another line and would call me back.

I am 100% sure they are aware of the problem and are fixing it. If I had to guess I would say they are most likely upgrading their SSL setup, since everyone is or should be.

i think it may be intermittent. we are still getting sales but are getting errors back on a lot of others.

How are you aware of the errors if you cant get in the admin?

Aaaaand my fucking join page is down :(

loads fine for me here :2 cents::2 cents:

what loads? my join page or the netbilling admin?

we don't use hosted join pages. the errors come back when we try to send transactions through the API.

I switched back to my old self hosted join pages and joins are coming in again, I am sure some are failing but no way to tell how many right now till the admin is back up. Maybe not then either.

Hi All,

NETbilling itself is not down. There is an outage with an interconnecting network backbone, Savvis/Centurylink. It has affected several websites on the internet including the NETbilling Admin Gateway from certain parts of the world. We are in touch with a team of engineers at the data center to get this issue resolved. It extends far beyond just NETbilling.

We will post here when resolved.

Thank you, Mitch Farber

In the meantime, if you have any questions, please contact [email protected] or call: 661-252-2456.

Thanks, Mitch

You can see Savvis network status here:

http://internetpulse.com

just got an email back from ben, thanks for the update.

Hi,

ok - sent you an email last week and no reply. I may be sending to an old address. Can you email me please with a good address?

Thanks, Mitch

Thanks for clarifying Mitch.

still down for me....

NETbilling Status


As many are already aware, there have been ongoing network routing issues today between Savvis (our host and network provider) and several of the major internet backbone providers. As a result, many websites and systems, including ours, have been unreachable to a large number of users throughout the day all over the internet. Savvis has indicated that they are performing network-wide maintenance activities to resolve the connectivity and latency issues, which unfortunately, is now directly impacting all NETbilling customers.

We will continue to work with the Savvis team, who are working under a master ticket, and we will bring our systems back up as soon as service is available to us and will notify you as well.

The latest update we have received is that Savvis expects all services to be back to normal by 2am PST.

You can monitor the networks here if you are interested.

If you have any questions, please email: [email protected].

Why do you guys not have multiple uplinks/backbone providers? It is crazy that you would rely on one single provider!

It seems CenturyLink is downplaying this as scheduled maintenance. Was Netbilling informed of this "scheduled maintenance"?

From Twitter:

You are doing it from inside NETbilling admin? Where is it/how's it called?

Good luck Mitch!

Quick question: Can you tell me how to read this page? http://internetpulse.com

its been over 24 hours now almost all down. no rebill nothing ....

Yes, joins halted last night and the admin is still unreachable. This is officially a nightmare.

I was just able to hit the admin portal for the first time today.

same here, yippeeee!

This situation was handled pretty bad. The email blast should have been sent an hour after you found out what is going on. Not 12 hours later.

Most of us have secondary billers and if we were notified properly this could have minimized the damage for us.

First time I am truly disappointed from Netbilling.

Let's be fair, Netbilling wasn't down, it just wasn't reachable for most of us. This was basically an ISP outage and I don't think most ISP customer feel they need to email all their customers everytime their ISP has an issue - The same reason Wal-Mart isn't likely to notify you of roadwork or traffic accidents that may affect your ability to shop at their stores. (not a perfect analogy, but similar logic)

Edit: Also, if we couldn't reach them, how are they supposed to reach us? The same issue keeping us out of our admins most likely kept them out unless they were onsite at the datacenter or something.

I doubt Mitch has all our emails in his phone and is able to thumb type notices to us individually. I'm just glad it's over.

up and running now

Netbilling is back up (and running very fast too!)

John is right - this wasn't Netbilling's fault, and even backup servers would not have made any difference because the problem was a network issue at a datacenter.

Their tech support was perfect. When we noticed the problem and contacted them, their tech support told me what the issue was right away and when we could expect it to be resolved right away.

As for any money lost due to not being able to bill, One thing that you can do next time is have multiple billing providers and then just switch your secondary biller to your primary. This is what we did.

cheers.

Hi all,

Email notification was sent out within the first hour. And again within 10 minutes if us coming back up at 4 this morning.
We are also updating our Call fire system for phone notifications so please be sure to email [email protected] any updated cell phone numbers that you want calls sent to if there is trouble.

This was the first time in 16 years that this type of issue occurred with Savvis. Their track record and ours are exceptional.

Thank you all for your business.

Rebills are all getting caught up now as well.

Mitch

Good to hear, handled very well.