February 2017

Intermittent errors with message queue
It seems likely that the intermittent errors were caused by a new setting we were testing for certain message queues, including some queues that were processing a high number of messages when the errors occurred. We have disabled this setting for these queues and will continue monitoring to see if the errors recur.
Feb 8, 09:17-20:57 UTC

January 2017

No incidents reported for this month.

December 2016

No incidents reported for this month.

November 2016

No incidents reported for this month.

October 2016

Issue with message queue
This incident has been resolved.
Oct 25, 09:37 - Oct 29, 00:23 UTC
Errors sending messages to Nexmo
The outage of Nexmo on Friday was related to the large distributed denial of service attack against Dyn (https://en.wikipedia.org/wiki/October_2016_Dyn_cyberattack). No additional DNS outages have been observed since Friday.
Oct 21, 17:26 - Oct 25, 09:40 UTC

September 2016

No incidents reported for this month.

August 2016

No incidents reported for this month.

July 2016

No incidents reported for this month.

June 2016

No incidents reported for this month.

May 2016

Issue with message queue
At approximately 00:10 UTC, one of our servers running RabbitMQ (the software that Telerivet uses internally to queue messages and other tasks) began experiencing very high CPU usage, very slow response times, and intermittent errors when queueing or dequeueing messages. During this time, messages were still able to be queued (with intermittent errors), and only 2% of API requests failed; however, the slow response times and intermittent errors from RabbitMQ caused the worker processes dequeuing messages to gradually fall further and further behind. Switching to a standby server in our RabbitMQ cluster did not resolve the issue. Eventually, we restarted the RabbitMQ process, at which time the CPU usage returned to normal, the intermittent errors stopped, and the worker processes quickly caught up. At this time, Telerivet has not yet identified a particular bug or configuration issue with RabbitMQ that caused this issue. In the next few days, we will be upgrading RabbitMQ to the latest release, as well as performing additional testing to try to reproduce the behavior in RabbitMQ outside of Telerivet's production environment.
May 11, 00:36-05:32 UTC

April 2016

Datacenter connectivity issue
This incident has been resolved.
Apr 12, 02:25 - Apr 13, 05:10 UTC

March 2016

No incidents reported for this month.

February 2016

No incidents reported for this month.

January 2016

[Scheduled] Scheduled datacenter migration
The migration to Google Compute Engine is complete.
Jan 16, 20:00-22:57 UTC
[Scheduled] Scheduled datacenter migration
The migration to AWS is complete, and all Telerivet services are currently operational.
Jan 2, 22:01-22:56 UTC
Datacenter connectivity issue
This incident has been resolved.
Dec 31, 12:44 - Jan 2, 17:02 UTC

December 2015

Datacenter connectivity issue
This incident is marked as resolved, because no network interruptions have been observed in the past several hours. In the meantime, we're actively working on improving our systems to increase reliability and reduce downtime in case of further DDoS attacks targeted at Linode.
Dec 30, 22:39 - Dec 31, 06:17 UTC
Datacenter connectivity issue
This incident has been resolved. Additional standby servers have been provisioned in a separate datacenter to reduce downtime in case Telerivet's primary data center has another extended outage in the future.
Dec 27, 05:17-20:22 UTC

November 2015

No incidents reported for this month.

October 2015

No incidents reported for this month.

September 2015

Network connectivity issue
Between 20:15 UTC and 20:18 UTC, and between 20:36 UTC and 20:49 UTC, there were periods of intermittent loss of network connectivity to Telerivet's servers due to network maintenance in Telerivet's data center. At this time the network issues have been resolved.
Sep 21, 20:42-21:23 UTC
Network connectivity issue
This incident has been resolved.
Sep 4, 21:30 - Sep 5, 00:35 UTC

August 2015

Web server connectivity issue
Telerivet's data center reported a network attack on another server in the data center. They have mitigated the issue and network connectivity has returned to normal. In total, the interruption to of the Telerivet web app lasted approximately 3 to 4 minutes.
Aug 5, 16:42-17:40 UTC

July 2015

Web server connectivity issue
The connectivity issue was identified as a hardware failure. The web server has been moved to new hardware to resolve the problem.
Jul 17, 05:52-07:52 UTC

June 2015

No incidents reported for this month.

May 2015

No incidents reported for this month.

April 2015

No incidents reported for this month.

March 2015

Hardware issue on API server
The hardware issue has been resolved.
Mar 28, 04:04-04:27 UTC
Network Connectivity Issues
Telerivet's data center reports that the packet loss issue has been resolved. In total, the packet loss resulted in a partial outage of Telerivet's services between 18:39 UTC and 19:11 UTC, with intermittent errors observed until 19:29 UTC.
Mar 25, 18:49-20:23 UTC
Database Interruption
The hardware issue has been resolved. All Telerivet services continue to be fully operational.
Mar 1, 02:26-03:18 UTC

February 2015

No incidents reported for this month.

January 2015

Network and Database Outage
The network issue was resolved and database connectivity has been restored.
Jan 31, 05:10-08:38 UTC

December 2014

No incidents reported for this month.

November 2014

No incidents reported for this month.

October 2014

No incidents reported for this month.

September 2014

No incidents reported for this month.

August 2014

No incidents reported for this month.

July 2014

No incidents reported for this month.

June 2014

No incidents reported for this month.

May 2014

No incidents reported for this month.