Due to the recent spam waves affecting the Fediverse, we’d like to open requests for comment on the use of automated moderation tools across Pawb.Social services.
We have a few ideas on what we’d like to do, but want to make sure users would feel comfortable with this before we go ahead with anything.
For each of these, please let us know if you believe each use-case is acceptable or not acceptable in your opinion, and if you feel like sharing additional info, we’d appreciate it.
1. Monitoring of Public Streaming Feed
We would like to set up a bot that monitors the public feed (all posts with Public visibility that appears in the Federated timeline) to flag any posts that meet our internally defined heuristic rules.
Flagged posts would be reported per normal from a special system-user account, but reports would not be forwarded to remote instances to avoid false-positives.
These rules would be fixed based on metadata from the posts (account indicators, mentions, links, etc.), but not per-se the content of the posts themselves.
2. Building of a local AI spam-detection model
Taking this a step further, we would like to experiment with using TensorFlow Lite and Google Coral Edge TPUs to make a fully local model, trained on the existing decisions made by our moderation team. To stress, the model would be local only and would not share data with any third party, or service.
This model would analyze the contents of the post for known spam-style content and identifiers, and raise a report to the moderation team where it exceeds a given threshold.
However, we do recognize that this would result in us processing posts from remote instances and users, so we would commit to not using any remote posts for training unless they are identified as spam by our moderators.
3. Use of local posts for non-spam training
If we see support with #2, we’d also like to request permission from users on a voluntary basis to provide as “ham” (or non-spam / known good posts) to the spam-detection model.
While new posts would be run through the model, they would not be used for training unless you give us explicit permission to use them in that manner.
I’m hoping this method will allow users who feel comfortable with this to assist in development of the model, while not compelling anyone to provide permission where they dislike or are uncomfortable with the use of their data for AI training.
4. Temporarily limiting suspected spam accounts
If our heuristics and / or AI detection identify a significant risk or pattern of spammy behavior, we would like to be able to temporarily hide / suppress content from the offending account until a moderator is able to review it. We’ve also suggested an alternative idea to Glitch-SOC, the fork we run for furry.engineer and pawb.fun, to allow hiding a post until it can be reviewed.
Limiting the account would prevent anyone not following them from seeing posts or mentions by them, until their account restriction is lifted by a moderator.
In a false-positive scenario, an innocent user may not have their posts or replies seen by a user on furry.engineer / pawb.fun until their account restriction is lifted which may break existing conversations or prevent new ones.
We’ll be leaving this Request for Comment open-ended to allow for evolving opinions over time, but are looking for initial feedback within the next few days for Idea #1, and before the end of the week for ideas #2 through #4.
You must log in or register to comment.
Appreciate the feedback so far, let me try to see if I can answer most / many of the questions:
What are the risks of #4?
Many users are worried about the risk of automated actions going wrong and not knowing what we mean with “pattern of spammy behavior.”
For how we would identify the pattern of behavior that would allow for automated actions, we would review any major spam wave, such as the one we’ve been experiencing over the past few days:
We would then identify any indicators we could use that are indicative of the known spam, and create a heuristic ruleset that would limit or suspend those accounts while targeting only those accounts actively engaging in the spam, not just referring to it. There are additional safeguards we can add, such as preventing rules being applied to users where the user is followed by someone on our instances.
For the risk of automated actions going wrong, if we were using a limit (not a suspend) then the account would be hidden from public view but could still be viewed if specifically searched by name, it would also suppress all notifications from that user unless they are followed by you. (e.g. if they messaged you out of the blue, you wouldn’t see it if you weren’t following them.)
If a suspend was used, the account would be marked for deletion from our instances but all follower relationships would immediately break (e.g. if you were following them, the system would automatically unfollow when they are suspended). Typically, we can restore data within 30 days, but follower relationships are typically unrecoverable. So long as rules are appropriately limited in scope to only target those with a lot of spam indicators, no false-positives should occur.
What about appeals?
For local users (anyone registered on furry.engineer and pawb.fun), all actions against your account (except reports) can be appealed. If you have a post removed or are suspended, all actions can be appealed directly to the admin team.
For remote users, we can remove restrictions on remote accounts if we receive an appeal from any of our users, or by the affected account directly. This can be done via email, or just through a DM to one of the admins who can pass it to the team.
Would the AI model have oversight?
Yes. Where the team believe the filter has flagged sufficient content appropriately and maintains no false-positives, we may promote a model or ruleset to allowing automated actions (limit / suspend).
We’ll keep an eye on the actions of each ruleset by reviewing the daily / weekly actions taken to ensure they meet the criteria and have not misidentified any users or content, and we’ll also start publicly tracking the statistics of the models / rulesets we create and use, including a count of false-positives or reversed decisions.
Will you notify users?
Due to limitations in Mastodon, we can only notify local users (users on furry.engineer or pawb.fun) when actions are taken against their account; This process happens automatically when your post is removed, or your account is warned, limited, or suspended.
There’s no easy way to notify remote users other than sending them a DM, but doing so could be seen as spammy or lead to inciting further abusive behavior by informing them of our activity. While we can have transparency with our users due to having an invite-only platform, other instances are frequently open-registration which can allow the abusive user to re-create an account to continue to harass our users. BUT, I’m open to suggestions on this.
hmm, on the last point: If it’s just a single user harassing then it shouldn’t be too much trouble if they re-create an account. The anti-spam system should flag them again if they keep harassing. If it’s a lot of bots then I would assume they already have methods to determine whether an account is suspended (like DM-ing each other maybe). Hence there wouldn’t be an advantage of not informing them of being suspended.
I might be completely wrong here and missing a key point as I don’t really know anything about Mastodon or spam prevention really but it just feels wrong to censor someone without them knowing.
If time is crucial you could inform people an hour/a day/etc. after their suspension.So, the issue lays in that there’s no technical way to notify the remote user (someone not on furry.engineer or pawb.fun) that they’ve been suspended on our end, without sending a message to them directly. If we suspend them on our end, that doesn’t per se suspend them on their end and they wouldn’t know that their messages were no longer reaching our users; They would still be able to message other users on their instance, and users on other instances, but not to our users.
We’re apprehensive about notifying remote accounts specifically because we don’t often know the moderation practices of the remote instance (to know if they’ll deal with it, or if they have open-registration allowing anyone to join without approval) and it may encourage further abusive behavior through ban evasions (creating new accounts on that instance or elsewhere to continue messaging) from the user being made aware that we’re no longer receiving their messages.
@crashdoom I’m generally against automated moderation having been shadowbanned on other platforms for no reason I can identify. These scripts are never infallable nomatter how well intentioned. A computer can be trained to recognise keywords but it can never understand context.
Having said that, I do appreciate the urgency to do something. If you do go ahead with it, I would ask the following:
- Make sure the user is informed of any action, never use shadowbans.
- Make sure there is easy access to human review in the event mistakes do occur.
@RavenLuni @crashdoom Yeah I agree. Automated moderation systems can cause a lot of problems when they ban or limit without human interaction.
If they do though, they need to inform the user of the actions performed, and there needs to be an easy way to appeal them, so they aren’t just baseless automated bans like on every mainstream service.
I think 1-3 are fine (since nothing really happens without a human involved), but 4 should come in after several months of testing the model to make sure its false positive rate is as close to 0 as possible.
I in general think that LLM/“AI” stuff is massively overblown when used for creating content, but when analyzing stuff, it’s much more reasonable to employ as a referral to humans to make the final decision.
I guess I’ve just been lucky in that I’ve not gotten any spam yet on masto…
I’m for automated flagging to help you with moderation but, especially with AI, you should review every action your flagger takes. There might be a big false positive percentage. And there should be a way for muted people to talk to an admin to get it resolved (maybe put a little link on the sidebar for that). Freezing posts so that no one can see them or comment on them seems fine even if it takes a day or so to resolve.
When someone gets muted they should get a clear message that informs them about this too. Like a whisper message maybe.
I like the idea, if I understood it correctly, that users can help train the AI by throwing typical spam at it. Maybe there could be a whole community just for that.
Other than that I’m fine with all points and I’m glad if you can future proof this server for protection against massive bot raids.
Thanks for your service :3@crashdoom I think automated moderation tools are potentially problematic unless they can be made to take into account the cultural norms of the person speaking. For example, there’s a traditional British food the name of which is a homophobic slur in American English. Another British slang term for a cigarette also falls foul of this. People who like this food have been auto-banned from other platforms for posting about it with no homophobic intent. I just don’t think intelligent mod tools are sufficiently capable to pick up from the speaker’s other posts or profile that what they said isn’t prejudiced because of who said it or that someone using the n-word is black and therefore the standards for whether the speech should trigger discipline could be radically different than if a white person said it.
If auto moderation is introduced, I think it’s important that the bot should message anyone it targets, tell them what its grounds were and how to appeal to a human if they think it was wrong
#1 makes sense; sure.
#2 seems fine to try but I am a little skeptical about the chances of success without domain knowledge. A Coral Edge TPU in particular feels quite unnecessary — most spam models are totally fine running on CPU. I am also a little surprised to see the first impression is to build rather than looking for existing local solutions.
#3 Sure, if it’s user by user opt in, that could be fine. I’d also ask — would false positives (flagged in an automated manner, reviewed by a human and found to be not spam) be entered as well to be trained on, or no?
#4 Seems reasonable, though I would hope that their posts would still be visible when directly viewing their profile page. I would also hope there is some mechanism in place such that automated techniques routinely misidentify a user, that they be exempted from this after ~2 times. I would also be curious to see some stats on this in transparency reports.
IMO this all seems basically fine, as long as no action is taken without human hands touching it. Flagging and the like are fine, I just don’t want to see conversations get broken since that’s literally the whole point of existing on fedi for me. I totally get trying to limit spam though, so it’s balancing it that’s important. I appreciate you actually caring about what people have to say.
@crashdoom #1 and #4 i am entirely for. regarding ai, i think that using a fully local model is a great idea, but as @jackemled points out, bias is something that ai models tend to pick up very easily from training data, so that’s something to be wary about. hopefully using only training data from this network’s spam reports and voluntary non-spam data should produce accurate results, but i’d still be especially cautious. i have some faith that it could work, though, and would be happy to be part of the non-spam training data if you decide to go that route.
I’m opposed to #4 on principle. ANY action taken against an account should ALWAYS be done by a person after direct review. It doesn’t matter if it can be fixed afterwards or not, you’re still potentially subjecting people to unfair treatment and profiling. You can have it notify moderators but the moderators should be the ones actually making the decision whether to limit an account for further investigation, not the auto-mod bot.
If you implement #4 as-is, I’m just flat-out not going to stick around.
EDIT: Also, I ran into an infinite loading bug when submitting this post.
I support 1, 2, 3, 4.
