MumbleBoxes Service Status

February 2017

Network Issues in Atlanta
Network issues in Atlanta have been resolved.
Feb 22, 01:49-04:54 UTC
Power Outage - Atlanta
This incident has been resolved.
Feb 15, 00:02-03:34 UTC

January 2017

No incidents reported for this month.

December 2016

Network issues in Atlanta
This incident has been resolved.
Dec 22, 02:06-05:37 UTC
Emergency Maintenance Tokyo
This incident has been resolved.
Dec 7, 00:45-02:19 UTC

November 2016

Server Issues: Panther
This incident has been resolved.
Nov 27, 22:33 - Nov 28, 18:15 UTC
Network issues in Atlanta
All servers are back up and all failover slaves are also back up.
Nov 12, 04:44-06:49 UTC

October 2016

No incidents reported for this month.

September 2016

Network issues in Dallas
The DDoS attack has been mitigated. Network behavior has returned to normal.
Sep 25, 00:41-03:32 UTC
Network Maintenance in Atlanta
This incident has been resolved.
Sep 22, 05:29 - Sep 24, 03:25 UTC
Mumble Server Maintenance
Our maintenance is completed.
Sep 16, 14:57-16:21 UTC
Network issues in Dallas
Network issues in Dallas have been resolved.
Sep 13, 21:09 - Sep 14, 00:33 UTC
Network issues in Atlanta
This incident has been resolved.
Sep 3, 21:15 - Sep 5, 16:10 UTC
Network issues in Dallas
This incident has been resolved.
Sep 5, 01:25-15:05 UTC
Network issues in Newark
Network issues in Newark were resolved and has been stable.
Sep 3, 04:39-05:50 UTC

August 2016

Network issues in Atlanta
Our datacenter has resolved these issues
Aug 30, 19:02-21:07 UTC
Network issues in Newark
Network issues in Newark appear to have been resolved.
Aug 26, 02:27-03:04 UTC
Server Disconnects
This has been resolved.
Aug 25, 00:53-01:01 UTC

July 2016

Network issues in Newark
This incident has been resolved.
Jul 10, 20:31 - Jul 11, 04:19 UTC

June 2016

Network Issue - fish.mumbleboxes.com
We have identified and resolved a network issue impacting fish.mumbleboxes.com The issue caused "split brain" issues, where you may not see all of the users connected to a server.
Jun 10, 00:22 UTC

May 2016

Network Issues in Atlanta
This incident has been resolved.
May 13, 00:56-15:36 UTC

April 2016

Issues with crow.mumbleboxes.com
crow has been up for about an hour. We have monitored it closely and appears stable.
Apr 29, 20:30-22:48 UTC
Network Maintenance in Tokyo
Network issues in Tokyo have been resolved.
Apr 19, 22:49 - Apr 20, 00:48 UTC
Network Issues in Atlanta
Network issues in Atlanta have been resolved.
Apr 19, 22:21 - Apr 20, 00:48 UTC
Restarting Mumble Servers
Maintenance is complete, all SSL certs have been updated.
Apr 3, 15:19-16:45 UTC
Network Issues in Dallas
This incident has been resolved.
Mar 19, 01:28 - Apr 3, 15:18 UTC

March 2016

Removed IPv6 Support
This incident has been resolved.
Mar 2, 00:38 - Mar 19, 01:28 UTC
DDoS in Dallas causing packetloss.
The packetloss issues in Dallas have been resolved.
Mar 18, 18:49-23:02 UTC
[Scheduled] Hardware Maintenance in Fremont
The scheduled maintenance has been completed.
Mar 18, 18:00-20:00 UTC
[Scheduled] Network Maintenance in Atlanta
The scheduled maintenance has been completed.
Mar 16, 00:01-01:00 UTC
[Scheduled] Network Maintenance in Atlanta
The scheduled maintenance has been completed.
Mar 7, 01:00-02:00 UTC
Emergency Maintenance: Node Reboots
All servers are back up.
Mar 2, 00:02-00:11 UTC
Network Maintenace in Atlanta.
Maintenace has been completed.
Mar 1, 23:09-23:36 UTC

February 2016

Security Update Maintenance
All security updates have been applied.
Feb 16, 22:52-23:28 UTC

January 2016

Security Update Maintenance
Maintenance is complete, thanks for sticking with us.
Jan 21, 22:06-22:27 UTC
Network Issues in Atlanta
Atlanta has been stable for a few hours, marking this as resolved.
Jan 4, 18:24-20:13 UTC
Network issues in Multiple Datacenters
Atlanta has been stable for about 18 hours. We are marking this resolved, but we will continue to monitor.
Dec 30, 22:05 - Jan 4, 16:55 UTC

December 2015

Network issues in Dallas
We are consolidating status updates into this single post, please see here: http://status.mumbleboxes.com/incidents/8g9xk91h7w6w
Dec 30, 21:53-22:06 UTC
Network Issues in Atlanta
We are consolidating status updates into this single post, please see here: http://status.mumbleboxes.com/incidents/8g9xk91h7w6w
Dec 30, 21:57-22:06 UTC
Network issues in Dallas
The attacks have been mitigated.
Dec 29, 19:03-22:10 UTC
Network issues in Dallas
This incident has been resolved.
Dec 26, 16:47 - Dec 27, 05:11 UTC
Network issues in Dallas
These issues are resolved.
Dec 25, 19:59 - Dec 26, 16:41 UTC
Network Issues in London
We have been monitoring network conditions in London and they have been stable for a few hours.
Dec 2, 23:23 - Dec 3, 02:28 UTC

November 2015

Database upgrade.
Database maintenance is complete.
Nov 17, 15:58-17:34 UTC
Network issues in Dallas
Network issues appear to have been resolved. We will continue to monitor.
Nov 15, 16:59-17:06 UTC
Network issues in Texas
Network issues in Dallas have subsided. We will continue to monitor for additional issues.
Nov 14, 03:00-03:05 UTC
Network Issues in Atlanta
Network issues in Atlanta have been resolved.
Nov 1, 16:36-21:23 UTC

October 2015

Packet loss in New Jersey
This incident has been resolved.
Oct 12, 22:08 - Oct 13, 02:06 UTC
Network packetloss in Dallas
Network issues have subsided and everything appears normal again. If you are having issues still, please open a support ticket.
Oct 10, 22:59-23:22 UTC

September 2015

Scheduled Maintenance in Fremont, Dallas and London.
The scheduled maintenance has been completed.
Sep 21, 15:21-21:32 UTC
Issues with Gorilla
Gorilla has been stable for the last half hour. Our fixes have resolved this.
Sep 17, 18:06-18:45 UTC
DNS outage
This incident has been resolved.
Sep 4, 10:32 - Sep 6, 00:21 UTC

August 2015

Network issues in Dallas
Network issues have subsided.
Aug 29, 21:37-22:37 UTC
Texas Network Issues.
We have experienced no additional packet-loss.
Aug 19, 02:55-03:23 UTC

July 2015

Falcon issues
We have resolved the issue with falcon.mumbleboxes.com and it is now online.
Jul 29, 14:03-14:06 UTC
Network Issues in Newark
At this time our upstream provider has resolved the connectivity issues and normal routing has been restored. If you have any additional questions or concerns please open a support ticket.
Jul 28, 20:41-22:02 UTC

June 2015

whale.mumbleboxes.com network issues
Whale has remained online and functional.
Jun 21, 01:20-16:46 UTC

May 2015

Fremont Outage
Power has been restored in our Fremont datacenter and servers are back online. We will follow up with our datacenter and provide a complete postmortem of this incident.
May 30, 02:02-06:41 UTC
Monkey Down
Monkey is back online, we will have a postmortem of what caused our monitoring to not automatically notify us of the outage. We sincerely apologize for the inconvenience this has caused, and absolutely understand that this is not acceptable. We will have more details on the steps we have taken to ensure this does not happen again in the postmortem.
May 23, 15:20-15:22 UTC

April 2015

Network issues in Dallas
Network issues in Dallas have been resolved.
Apr 22, 03:25-03:38 UTC

March 2015

Network issues: Atlanta
This incident has been resolved.
Mar 29, 21:06 - Mar 30, 00:19 UTC
Rolling restarts for SSL updates.
All servers have been restarted.
Mar 19, 19:35-19:53 UTC
SSL Certificate Expiration
We are marking this incident as resolved as it has been over 2 weeks since we updated the certs.
Mar 4, 17:04 - Mar 19, 19:23 UTC
[Scheduled] Maintenance in Japan: 2015-03-03 07:00 PM GMT
Scheduled maintenance in Tokyo has been completed.
Mar 3, 19:00-19:59 UTC
Quick Maintenance Required: London and Dallas
This incident has been resolved.
Mar 1, 18:00 - Mar 2, 14:20 UTC
Network Issues in Japan
The network issues have been resolved.
Mar 1, 17:23-17:50 UTC

February 2015

No incidents reported for this month.

January 2015

Packetloss in Dallas
Packetloss has been resolved.
Jan 28, 19:28-20:00 UTC
Rolling server restarts (security fixes)
Rolling restarts are complete. Thanks for your patience :)
Jan 27, 17:15-17:39 UTC

December 2014

Packetloss in Dallas
Packetloss has been resolved, we are continuing to monitor to ensure it remains stable.
Dec 27, 20:15-20:24 UTC
Network Issues in Texas
Network issues for bear have been completely resolved.
Dec 3, 04:52-05:44 UTC

November 2014

Issues with eagle.mumbleboxes.com
These issues have been resolved. They were caused by hardware failure in one of the pieces of network gear at our datacenter.
Nov 10, 17:07-20:56 UTC

October 2014

Network issues in Dallas
This incident has been resolved.
Oct 1, 19:49 - Oct 4, 19:31 UTC
Packet loss in Texas
This incident has been resolved.
Oct 1, 03:06-16:19 UTC

September 2014

Urgent Rolling Reboots
This incident has been resolved.
Sep 24, 20:28-21:24 UTC
Network Issues: London, England
This incident has been resolved.
Sep 24, 19:47-20:19 UTC
Website Issues
This issue has been resolved.
Sep 17, 19:59 - Sep 18, 00:11 UTC
Mumble version upgrades
Upgrades are complete.
Sep 8, 15:51-15:55 UTC

August 2014

No incidents reported for this month.

July 2014

Critical Security Updates Underway
This incident has been resolved.
Jul 18, 19:55 - Jul 19, 08:03 UTC
We are investigating an issue with lion.mumbleboxes.com
This incident has been resolved.
Jul 17, 08:18-08:24 UTC

June 2014

Resolving Latest OpenSSL Security Advisory
All servers have been updated.
Jun 5, 17:13-23:19 UTC
New Jersey Network Issues
This incident has been resolved.
Jun 2, 21:52 - Jun 3, 02:17 UTC

May 2014

[Scheduled] Network Maintenance in Newark: May 11, 2014
The scheduled maintenance has been completed.
May 11, 06:00-08:00 UTC
Dallas Network Issues
This incident has been resolved.
May 7, 21:14-21:50 UTC

April 2014

Newark Network Issues
Our datacenter had indicated that these attacks have been mitigated and the issues have been resolved.
Apr 29, 20:15-20:55 UTC
Fremont Network Issues - Main website issues.
Our datacenter had indicated that these attacks have been mitigated and the issues have been resolved.
Apr 29, 18:56-20:55 UTC
Server Restarts for Upgrades
All disconnect hiccups are now complete :) Thanks for sticking with us.
Apr 19, 14:17-15:05 UTC
Newark NJ Network Issues
All servers are back up and functional. We will be following up with the datacenter and posting a postmortem.
Apr 13, 03:44-08:01 UTC
Network issues - falcon
Issue has been resolved
Apr 12, 20:19-20:52 UTC
Server Upgrades / Rolling Reboots
All servers have completed reboots, and no further reboots are expected. Thank you all for hanging in there. We will be updating with a post-mortem in a few moments.
Apr 8, 16:05-20:39 UTC

March 2014

Network Issues in Atlanta
This issue has been resolved by our upstream provider. There may be intermittent packet loss or connectivity problems lingering but Atlanta is now accessible.
Mar 31, 18:26-19:21 UTC
[Scheduled] Network Maintenance in Tokyo: March 17, 2014
The scheduled maintenance has been completed.
Mar 16, 18:00-18:25 UTC
Network maintenance in Atlanta.
Maintenance is complete.
Mar 5, 04:21-05:03 UTC

February 2014

Resolved issue with falcon.mumbleboxes.com
We have found and resolved an issue with falcon.mumbleboxes.com
Feb 21, 07:40 UTC
Atlanta Network Packetloss
This incident has been resolved.
Feb 21, 01:48-07:18 UTC
[Scheduled] London Network Maintenance
The scheduled maintenance has been completed.
Feb 20, 22:55 - Feb 21, 00:00 UTC
DDoS Attack Against Datacenter.
The attack was mitigated at 5:07PM EST. Services are behaving normally. The attack was mitigated in about 7 minutes.
Feb 12, 22:03-22:39 UTC
Emergency Update
Updates have been completed.
Feb 12, 21:32-22:01 UTC
Dallas Packetloss
DoS attacks have been successfully mitigated.
Feb 12, 17:58-18:32 UTC
Packet loss in Dallas
Packet loss has subsided. We have asked our datacenter for an update. Of what the cause is.
Feb 11, 04:22-04:31 UTC
Packetloss in Dallas
Packetloss has disapeared. We will continue to monitor.
Feb 9, 02:28-02:35 UTC
Eagle Issues
We have restarted the server and it is back up and taking connections. We will investigate the cause.
Feb 2, 18:07-18:13 UTC

January 2014

Website Updates - Deployment in progress
Website has been updated!
Jan 20, 00:12-00:19 UTC
Packetloss in Dallas
Packetloss in Dallas has subsided, however we will continue to monitor.
Jan 10, 06:42-07:02 UTC
falcon.mumbleboxes.com - Connection issues
We have flipped falcon to the failover backup. It is back online. The old master has been restarted and is also online as the new failover backup.
Jan 10, 06:25-06:30 UTC
London Datacenter Issues
At this time the issue has been resolved. If you are still experiencing any issues, please feel free to open a support ticket.
Jan 10, 05:24-06:08 UTC
Idle User Script Issues
We have identified the issue that caused the idle worker to stall and have deployed a fix.
Jan 10, 00:06-00:55 UTC
Network issues - Fremont
This issue should be resolved at this time and networking connectivity should be back to normal. If you are still seeing issues please open a support ticket so that we can investigate further.
Jan 3, 01:49-02:09 UTC
Upgrading Webservers
Webservers have been upgraded, all systems operational.
Jan 2, 20:40-23:54 UTC