cross-posted from: https://news.abolish.capital/post/12444
I pushed an update tonight that will start filling out the the post body with the full article text on many more feeds.
The next goal after this is to handle feeds that do not provide full text and pull them straight from the publication URL. I’ll have to be careful to not be to aggressive when doing that task to avoid getting placed on a black list.
I should maybe clerify, by “black list” I mean, by the publication lol. There are a few publications I have to access via a VPS now because the rate at which I was pulling their RSS feed was much too frequent, so they blocked my IP address from accessing their site. Pulling a single RSS feed every 30 to 60 minutes is reasonable, but if I get a block of updates, I want to be careful not to then hit their server another 5 or 6 times in a matter of seconds or minutes. I could trigger DDOS protections and get the RSS bot blocked.
If consolidating news into a single feed for broad distribution is going to get me in any real trouble, then we’re all doomed!
I know. I was thinking about the current government lists, additionally.