mastodon.xyz suspension on July 5, 2023

On July 5, 2023 from 19:32 CEST, mastodon.xyz became progressively unavailable to all users. The service was restored the following day at 20:26 after a downtime of just over 24 hours.

As this was a highly unusual event, I’m writing this article to explain what happened.

This is a translation of the original article, in French.

Context

mastodon.xyz was created on the 1st of April 2017. Most of the registered users, around 24,000, arrived before 2019, then registrations were closed until November 2022, as it had become far too time-consuming to manage the massive creation of bots and spam accounts on the instance.

The reopening of the registrations in November 2022 significantly increased the moderation work, from a few reports per week to a few dozens. Also, while most of the behavior reported in the past was not very dangerous, there are now a substantial number of accounts scamming, spamming or publishing illegal content, which are of course suspended when reported.

Since moderation work is done in my spare time (I’m currently the only truly active administrator and moderator of the instance), there usually is a delay between a report being made and its processing, sometimes up to a few days.

I closed the instance registrations again a few weeks ago, a little tired of having to spend hours checking and deleting illegal content or spam.

Abuse report

Two hours before the suspension, I received an e-mail from Hetzner, the hosting provider of mastodon.xyz. It was an abuse report, a fairly standard procedure for reporting illegal content. Hetzner received a report, checked it and forwarded it to me, asking me to delete the content at the provided URL.

The content in question is unfortunately something that has become very common in recent months: CSAM (child sexual abuse material), generally AI-generated. I also noted that I had indeed received several reports via Mastodon over the past few days for this same account, which I had not yet dealt with. I deleted the account, took the opportunity to process the rest of the remaining reports, notified Hetzner of the resolution and got on with my day.

Suspension

At 19:32, the first problems were reported by my monitoring system, but I didn’t see it right away. I realized the seriousness of the situation around 21:00.

The instance servers were working fine, but the domain was basically gone. At this point, browsers displayed an error of this kind.

Using the dig tool, we can easily check what’s going on. The DNS resolvers of the .xyz top-level domain registry return a NXDOMAIN code, which means that the domain does not exist.

To understand why, we can use a whois service to obtain, in this case, information about a domain name. Here’s an extract of what we get.

Domain Name: MASTODON.XYZ
...
Registry Expiry Date: 2024-04-01T23:59:59.0Z
Registrar: OVH
Registrar IANA ID: 433
Domain Status: clientTransferProhibited https://icann.org/epp#clientTransferProhibited
Domain Status: clientDeleteProhibited https://icann.org/epp#clientDeleteProhibited
Domain Status: serverHold https://icann.org/epp#serverHold

We can see that the domain hasn’t expired, so that’s not the problem. What’s interesting is the last line, with the serverHold status. This means that the domain has been suspended by the registry of the xyz TLD, and not by the registrar, OVH in my case, which had nothing to do with it.

The most likely scenario at this point is that the registry received the same report as Hetzner earlier, and immediately reacted by suspending the domain.

Contacting support

The only way to solve the problem seems to be a form on the registry website which allows you to unsuspend your domain in three simple steps.

In truth, the steps aren’t very simple, as the information given is incorrect, but they’re also not very fast either, since the announced timeframe is a maximum of 48 hours, which looks like a lot to me for a suspension without prior notice.

In any case, all we can do now is wait.

Communication

Normally, when the instance has problems or maintenance is in progress, I display an error message explaining the situation on the homepage. This is obviously impossible in this case.

I posted a few tweets in the evening, and also left a few messages on the Mastodon project Discord server. But that’s not enough and I needed to warn as many users of the instance as possible, by e-mail for example.

As I still had access to the servers, I extracted the list of users active in the last month, wrote a few lines of JavaScript and an e-mail in English and French to send to the 3200 users I extracted. And then, because it would had been too simple otherwise, a new problem appeared.

Normally, mastodon.xyz e-mails (registrations, notifications, etc.) are sent from the noreply@mastodon.xyz address. Obviously, I could not do that, so I wanted to send them from another of my domains, kinrar.io.

Mailgun is a company that provides email services for developers, which I use for Mastodon, and which I was trying to use here too.

Seeing the amount of emails I was sending from a new domain, their system blocked my sendings.

Fortunately, their support is available and relatively responsive, but ultimately terribly inefficient in my case. After five and a half hours and a dozen messages, it’s eight o’clock and the answer is clear: they’ll unblock the sending of e-mails when the instance comes back online, as they need to check the privacy policy. My attempts to redirect to other instances with the same policy, to temporarily host the instance on a sub-domain of kinrar.io and to explain the situation were to no avail.

Here is their answer:

After reviewing the account in detail, we have determined that the sending limitation will remain in effect temporarily due to the lack of required documentation for the domain mastodon.xyz. Once it is up and running please update this ticket for another review.

Great. I ended up sending the rest of the mails in the afternoon with my personal mail server, but the result is that a lot of them were classified as spam.

Here’s the email sent to active users of the instance:

Hi TheKinrar,

mastodon.xyz has been unavailable since yesterday (July 5) around 19:00 CEST. Our domain (mastodon.xyz) was suspended by the register of the xyz top-level domain, causing this downtime.

It looks like the registry acted upon a report they received about illegal content from one of the instance users, without contacting us before. It also seems that the content had actually been removed before the suspension occured. With the influx of new users, some people try to abuse the instance, and we suspend them when reported. This is still speculation for now though, some of this could be wrong.

We sent a message to the register, but they seem to be quite slow to process requests. We expect a fix in 48 hours maximum. Until then, there is nothing more we can do.

Please note that when the instance comes back up, it will be lagging behind other instances of the fediverse, so you should expect a few hours of slowdowns while it catches up.

This is the longest unavailability of mastodon.xyz since it was created, and we are doing our best to fix this issue as soon as possible.

See you soon on mastodon.xyz

Fix

At 20:11, I finally received a reply from the registry, confirming what I explained earlier: the domain had indeed been suspended due to the content published by one of the instance users, content which, ironically, had been deleted minutes before the suspension, following Hetzner’s email. On the positive side, they explain that they have added the instance to their list of false positives, which will prevent further suspensions in the future.

Here’s the original text of their response:

Hello,

Thank you for your response. I’m sure you can appreciate that we as the registry operator want .xyz to stay a safe and reputable namespace. To do this, we collect data from over 100 highly respected abuse feeds, including URIBL, Google, and Phishlabs, and temporarily suspend domains that we believe have been hacked or are being used for malicious purposes. As a result, it is well established in the industry that .xyz is a safe namespace and your visitors can feel confident they are going to a reputable website when they see your website link.

This domain has been flagged for violating our anti-abuse policies, specifically child sexual abuse material by one of our cybersecurity partners.

I have unsuspended this domain and added this to a false positive list to prevent further suspension.

Please remove the URL containing CSAM

URL: hxxps://mastodon.xyz/[…]

Let me know if there is anything else I can help you with.

About a quarter of an hour later, the instance was up and running again. A few slowdowns occurred, as expected, as all the backlog accumulated over the past 24 hours had to be cleared. This mainly involves receiving messages and interactions sent by other instances.

What’s next

Now that everything’s back to normal, I’d like to take this opportunity to reassure anyone who may be worried about the future of mastodon.xyz. There is no intention on my part to close down the instance in the years to come. It’s just the opposite. I’ve got a lot of great memories on this instance, and above all I have a great responsibility to the thousands of people who have used it and still do. I hope to be able to maintain it for a very long time.

On this subject, if you wish, you can contribute to the hosting costs of the instance (currently around 150 to 200 €/month on average) via Patreon, Liberapay or PayPal. Some people have been supporting the instance for years, and I’d like to thank them. I couldn’t have maintained the instance without their help over all these years!

It’s not decided yet, but a status page for the instance will soon be deployed on another domain name, to ensure communication in the event of similar issues.
I’m also likely to be expanding the moderation team in the coming weeks to ensure faster processing of reports.

Thank you to those who sent supportive messages. I didn’t have much time to reply, but I read everything, and your messages made me very happy and were very motivating! See you soon on Mastodon!