mastodon.xyz suspension on July 5, 2023
On July 5, 2023 from 19:32 CEST, mastodon.xyz became progressively unavailable to all users. The service was restored the following day at 20:26 after a downtime of just over 24 hours.
As this was a highly unusual event, I’m writing this article to explain what happened.
This is a translation of the original article, in French.
mastodon.xyz was created on the 1st of April 2017. Most of the registered users, around 24,000, arrived before 2019, then registrations were closed until November 2022, as it had become far too time-consuming to manage the massive creation of bots and spam accounts on the instance.
The reopening of the registrations in November 2022 significantly increased the moderation work, from a few reports per week to a few dozens. Also, while most of the behavior reported in the past was not very dangerous, there are now a substantial number of accounts scamming, spamming or publishing illegal content, which are of course suspended when reported.
Since moderation work is done in my spare time (I’m currently the only truly active administrator and moderator of the instance), there usually is a delay between a report being made and its processing, sometimes up to a few days.
I closed the instance registrations again a few weeks ago, a little tired of having to spend hours checking and deleting illegal content or spam.
Two hours before the suspension, I received an e-mail from Hetzner, the hosting provider of mastodon.xyz. It was an abuse report, a fairly standard procedure for reporting illegal content. Hetzner received a report, checked it and forwarded it to me, asking me to delete the content at the provided URL.
The content in question is unfortunately something that has become very common in recent months: CSAM (child sexual abuse material), generally AI-generated. I also noted that I had indeed received several reports via Mastodon over the past few days for this same account, which I had not yet dealt with. I deleted the account, took the opportunity to process the rest of the remaining reports, notified Hetzner of the resolution and got on with my day.
At 19:32, the first problems were reported by my monitoring system, but I didn’t see it right away. I realized the seriousness of the situation around 21:00.
The instance servers were working fine, but the domain was basically gone. At this point, browsers displayed an error of this kind.
dig tool, we can easily check what’s going on. The DNS resolvers of the
.xyz top-level domain registry
NXDOMAIN code, which means that the domain does not exist.
To understand why, we can use a
whois service to obtain, in this case, information about a domain name.
Here’s an extract of what we get.
Domain Name: MASTODON.XYZ ... Registry Expiry Date: 2024-04-01T23:59:59.0Z Registrar: OVH Registrar IANA ID: 433 Domain Status: clientTransferProhibited https://icann.org/epp#clientTransferProhibited Domain Status: clientDeleteProhibited https://icann.org/epp#clientDeleteProhibited Domain Status: serverHold https://icann.org/epp#serverHold
We can see that the domain hasn’t expired, so that’s not the problem. What’s interesting is the last line,
serverHold status. This means that the domain has been suspended by the registry of the
and not by the registrar, OVH in my case, which had nothing to do with it.
The most likely scenario at this point is that the registry received the same report as Hetzner earlier, and immediately reacted by suspending the domain.
The only way to solve the problem seems to be a form on the registry website which allows you to unsuspend your domain in three simple steps.
In truth, the steps aren’t very simple, as the information given is incorrect, but they’re also not very fast either, since the announced timeframe is a maximum of 48 hours, which looks like a lot to me for a suspension without prior notice.
In any case, all we can do now is wait.
Normally, when the instance has problems or maintenance is in progress, I display an error message explaining the situation on the homepage. This is obviously impossible in this case.
I posted a few tweets in the evening, and also left a few messages on the Mastodon project Discord server. But that’s not enough and I needed to warn as many users of the instance as possible, by e-mail for example.
Normally, mastodon.xyz e-mails (registrations, notifications, etc.) are sent from the firstname.lastname@example.org address. Obviously, I could not do that, so I wanted to send them from another of my domains, kinrar.io.
Mailgun is a company that provides email services for developers, which I use for Mastodon, and which I was trying to use here too.
Seeing the amount of emails I was sending from a new domain, their system blocked my sendings.
Here is their answer:
After reviewing the account in detail, we have determined that the sending limitation will remain in effect temporarily due to the lack of required documentation for the domain mastodon.xyz. Once it is up and running please update this ticket for another review.
Great. I ended up sending the rest of the mails in the afternoon with my personal mail server, but the result is that a lot of them were classified as spam.
Here’s the email sent to active users of the instance:
mastodon.xyz has been unavailable since yesterday (July 5) around 19:00 CEST. Our domain (mastodon.xyz) was suspended by the register of the xyz top-level domain, causing this downtime.
It looks like the registry acted upon a report they received about illegal content from one of the instance users, without contacting us before. It also seems that the content had actually been removed before the suspension occured. With the influx of new users, some people try to abuse the instance, and we suspend them when reported. This is still speculation for now though, some of this could be wrong.
We sent a message to the register, but they seem to be quite slow to process requests. We expect a fix in 48 hours maximum. Until then, there is nothing more we can do.
Please note that when the instance comes back up, it will be lagging behind other instances of the fediverse, so you should expect a few hours of slowdowns while it catches up.
This is the longest unavailability of mastodon.xyz since it was created, and we are doing our best to fix this issue as soon as possible.
See you soon on mastodon.xyz
At 20:11, I finally received a reply from the registry, confirming what I explained earlier: the domain had indeed been suspended due to the content published by one of the instance users, content which, ironically, had been deleted minutes before the suspension, following Hetzner’s email. On the positive side, they explain that they have added the instance to their list of false positives, which will prevent further suspensions in the future.
Here’s the original text of their response:
Thank you for your response. I’m sure you can appreciate that we as the registry operator want .xyz to stay a safe and reputable namespace. To do this, we collect data from over 100 highly respected abuse feeds, including URIBL, Google, and Phishlabs, and temporarily suspend domains that we believe have been hacked or are being used for malicious purposes. As a result, it is well established in the industry that .xyz is a safe namespace and your visitors can feel confident they are going to a reputable website when they see your website link.
This domain has been flagged for violating our anti-abuse policies, specifically child sexual abuse material by one of our cybersecurity partners.
I have unsuspended this domain and added this to a false positive list to prevent further suspension.
Please remove the URL containing CSAM
Let me know if there is anything else I can help you with.
About a quarter of an hour later, the instance was up and running again. A few slowdowns occurred, as expected, as all the backlog accumulated over the past 24 hours had to be cleared. This mainly involves receiving messages and interactions sent by other instances.
Now that everything’s back to normal, I’d like to take this opportunity to reassure anyone who may be worried about the future of mastodon.xyz. There is no intention on my part to close down the instance in the years to come. It’s just the opposite. I’ve got a lot of great memories on this instance, and above all I have a great responsibility to the thousands of people who have used it and still do. I hope to be able to maintain it for a very long time.
On this subject, if you wish, you can contribute to the hosting costs of the instance (currently around 150 to 200 €/month on average) via Patreon, Liberapay or PayPal. Some people have been supporting the instance for years, and I’d like to thank them. I couldn’t have maintained the instance without their help over all these years!
It’s not decided yet, but a status page for the instance will soon be deployed on another
domain name, to ensure communication in the event of similar issues.
I’m also likely to be expanding the moderation team in the coming weeks to ensure faster processing of reports.
Thank you to those who sent supportive messages. I didn’t have much time to reply, but I read everything, and your messages made me very happy and were very motivating! See you soon on Mastodon!