We are continuing to monitor for any further issues.
Posted Jul 17, 2026 - 12:13 AEST
Monitoring
On 17 July 2026, customers on the AU platform experienced call timeouts and failures on PSTN calls. Some also saw delayed or failed SIP device registration. Today, a brief follow-on outage occurred as the CEPH cluster completed its recovery from yesterday's incident.
Cause A log rotation script that should have deleted transcribed voicemail messages stopped running. Messages accumulated on our shared storage layer (CEPH) until it neared capacity and began blocking writes. This starved other services sharing that storage, including the SIP proxies that set up PSTN calls.
What is CEPH? CEPH is a distributed storage platform that unifies block, file, and object storage. It pools disks across ordinary servers into a fault-tolerant cluster: if a disk or server fails, data is recomputed and redistributed from surviving copies automatically. That resilience comes at a cost — CEPH can become a bottleneck at scale.
What we're doing about it Since May we've been separating core voice and network services from CEPH: removing CEPH as the backing store for the voice network entirely, and upgrading from mechanical drives to server-class SSDs. That project is in its final weeks. Once complete, storage growth from voicemail or recordings will no longer be able to affect call setup or SIP services.
On a brighter note Today I released Voice AI agents, running on our own private GPU cluster in Sydney, and our desktop client now includes a full call centre system, with video conferencing across desktop and mobile scheduled for release later this month.
Posted Jul 17, 2026 - 12:13 AEST
Identified
The issue has been identified and a fix is being implemented.
Posted Jul 17, 2026 - 11:15 AEST
This incident affects: SIPcity Calling Platform (Network) and SIPcity Cloud PBX.