As part of the next XMB version in development, I made changes to help with the new pricing structure of Google's reCAPTCHA service.
In our original implementation, the captcha was displayed as soon as anyone visits the registration page. This causes unwanted traffic between bots
and the Google API.
Going forward, there will be an introduction page for registration. This means the reCAPTCHA gets pushed back to the 2nd page and won't be
visited by bots quite so frequently.
There is also a future change regarding keys "to be migrated to a Google Cloud projected by the end of 2025." It is my next priority to
learn how that affects us here.miqrogroove - 3-3-2025 at 02:59 AM
What I've learned so far is that it's not just about the keys. I migrated one of the 2 keys used here and that transition is seamless.
After that, there are a bunch of steps for actually switching APIs from the "siteverify" to the "enterprise" version. Here's
how that might go:
So this looks like a need for more settings, a way to switch between the two API implementations, and a lot of testing after it's ready.miqrogroove - 3-4-2025 at 07:43 PM
I'd rather avoid captchas altogether, they just annoy people while doing little to actually prevent spammers.
My proposal is to go for some user-configurable Q&A,. Then again, good luck educating admins to bother setting it up properly so it's both
accessible enough for users and not too easy for spammers. (
I remember setting up a phpbb board over 4 years ago that I've long since left, and there was a 'sortable q&a' feature that had
minimal JS and made you put different things into different boxes:
I think there's maybe been one spam account that registered in the whole time of its existence, and while it's... obnoxiously opinionated,
to say the least, it works for the purpose of that specific forum.
In other words, configure something like this correctly, and you've got yourself something that works with minimal JS and tracking things -
therefore works even under 'strained conditions'. You could expand on the idea a bit more than phpbb did which I think only allows for the
one question - as long as the suggested defaults are strong I think admins should be okaymiqrogroove - 3-19-2025 at 10:32 PM
If there's an open source solution that you know about, that's possible. This isn't a wheel that we should try to reinvent.miqrogroove - 5-4-2025 at 02:45 PM
These different vendors have interesting free features. I'm just not sure yet if we need to support more than one.flushedpancake - 5-11-2025 at 04:22 AM
These different vendors have interesting free features. I'm just not sure yet if we need to support more than one.
I've seen numerous sites using that one. Might be worth adding since it's probably the one I encounter most after reCAPTCHA, and it seems to
have especially come about ever since the changes to the API.miqrogroove - 6-26-2025 at 08:21 AM
I'm reading more about these and have some new thoughts:
reCAPTCHA doesn't specifically require a migration to the new API. What they are really going to require is a new pricing structure. So
I'm seeing a need to install the alpha version of XMB and find out if those updates will cut the reCAPTCHA traffic counts to an affordable level.
If not, I would be inclined to move away from Google on this feature rather than implement a new API.
Cloudflare seems more committed to offering a free tier and eliminating user puzzles. This is my fallback plan.
hCaptcha is a little weird in the details. They offer a free tier, but they want $140 per month for passive verification. This puts them at a huge
disadvantage.flushedpancake - 6-28-2025 at 06:55 AM
I've never had issues with the cloudflare verification thingy. It seems like it might be the way to go. miqrogroove - 6-28-2025 at 01:15 PM
I'm monitoring the bot traffic this morning and the results are interesting. Even though the bots are requesting captcha at a rate well in
excess of 10,000 per month, the success rate is much lower. The XMB server is only seeing the verification/assessment process at a rate of about
3,000 per month.miqrogroove - 6-28-2025 at 04:53 PM
The last thing to tackle for reCAPTCHA specifically is the new settings offered by Google. During setup, on the reCAPTCHA Classic site, it asks for
"Score based (v3)" or "Challenge (v2)" type protection. Currently, we only support v2. And then it asks for Checkbox vs.
Invisible, and we're not set up for the invisible mode. I also suspect there's no way for XMB to check which settings were used on the
Google side of the API.
On the Google Cloud site, it doesn't ask for those things during setup. In step 2, there's a slider labeled, "Will you use
challenges?"
I don't even know which combinations of these different sites and APIs are currently compatible with XMB. We will need to do some experiments
and offer some instructions.flushedpancake - 6-28-2025 at 08:59 PM
I'm monitoring the bot traffic this morning and the results are interesting. Even though the bots are requesting captcha at a rate well in
excess of 10,000 per month, the success rate is much lower. The XMB server is only seeing the verification/assessment process at a rate of about
3,000 per month.
Any specific patterns with the success rate? Countries, browser agents etc?
Could be worth looking into for the sakes of analytical purposes. I'm a little curious if anything crops up in particular.miqrogroove - 6-29-2025 at 01:55 AM
I don't want to advertise the details too much so let's just say the latest changes are having a big impact on the captcha traffic.
Officially, Google only publishes daily numbers, so after a few days I can share a screenshot of what happened in the daily trend.
Also remember, these changes tend to be temporary. The bots are always evolving so what works today isn't going to work years later.
I'm also weary of Google's intent to either monetize the bot traffic or "fail open" when quotas are exceeded. That's bad
policy. I will continue to evaluate the idea of offering the Cloudflare alternative.flushedpancake - 6-29-2025 at 09:09 AM
Understandable, don't want to give people ideas I suppose.
One thing about reCAPTCHA I have noticed that it treats you much more leniently if you're using latest Chrome/Edge/Safari and are logged into a
Google account (and I presume it uses the data from that to determine whether to give a challenge or not). This, I'll be honest, is quite the
concern for user privacy...
Personal anecdote - there are times I have recalled having to switch to my phone simply to fill out a reCAPTCHA because it's temporarily
"blocked my network" - but then suddenly doesn't care when it's iOS Safari...
CloudFlare on the other hand doesn't seem to discriminate in this way.miqrogroove - 6-30-2025 at 08:02 PM
Captcha requests on 6/26: 1,695
Captcha requests on 6/29: 2flushedpancake - 7-1-2025 at 03:23 PM
LOL nice, literally just from moving it to the other page?
Btw, two of the links in the who's online thing for looking up IPs are broken, IIRCmiqrogroove - 7-1-2025 at 06:31 PM
Please consider ipinfo; cloudflare uses them for geolocation ip data
community.ipinfo.io/t/ipinfo-is-the-ip-geolocation-data-provider-of-cloudflare/6841
In an ideal world, forum admins could add 3 or 4 ip data providers of their own choice in the admin panel as providers seem to change their access or
remove altogether.flushedpancake - 7-2-2025 at 11:17 AM
Even the labels seem wrong. W and T seem to be the only ones that have meaning.
I guess you could always send in a github pull request. I'm cleaning up that bbcode doc file atm.miqrogroove - 7-2-2025 at 11:17 PM
I might have to delete the traceroute link. Have you found any alternative?flushedpancake - 7-3-2025 at 12:13 AM
dnsquery.org seems to work and has no rate limit.
Despite it not having any visual work done to it in about 10 years, it seems like it is still maintained to some extent, and the domain was last
renewed earlier this year.miqrogroove - 7-3-2025 at 12:48 AM
That one is too slow and doesn't show reverse lookup names. I'd rather get rid of the link if there aren't any good ones.lottos - 7-3-2025 at 01:42 AM
The reCAPTCHA Classic admin dashboard threw a flag for the change in traffic stats.
"We detected that your site is verifying reCAPTCHA passed solutions less than 50% of the time. This could indicate a problem with your
integration with reCAPTCHA. Please see our developer site for more information."
I went through the first few steps of the XMB registration process again and confirmed it is working normally. So the admins using this feature can
expect that flag to show up after switching to XMB v1.10.Jason Fairmount - 7-3-2025 at 11:59 PM
Not to sidetrack, but is email OTP verification worth looking into?
Edit: Apparently password/email verification has been an option since forever. I had never noticed this; on my board we always used 1 IP a day and a
hack (the additional questions mod). I was thinking of the 6 digit random numbers, but now that doesn't make sense. Carry on; I was never
here.flushedpancake - 7-4-2025 at 03:06 AM
I'm pretty sure that's already a thing anyway: the verification used here sends a random password, at least...
Unless you mean something like sending a one time verification code to the email address associated with the account when logging in from a new
location...
Also I need to stop using elipses.miqrogroove - 7-4-2025 at 02:52 PM
Registration script now provides soft errors for the required fields: Username, password, and e-mail address. When at least one of those is invalid,
the profile creation step will save all inputs and refresh with a message at the top. This enables the user to continue registration without going
back to the captcha step after every mistake.miqrogroove - 7-6-2025 at 04:06 PM
Switching to an "invisible" site key didn't cause any problems. In fact, the v2 API doesn't seem to care what the console
settings are, nor does it inform the client which type of key it is. It just follows the rendering in our templates and shows the checkbox.
Adjusting the template to use a different element switches the captcha to invisible mode.
Now I need to look at the siteverify API v3 and figure out if we need to support that too, and how is it any different from the Enterprise API.miqrogroove - 7-6-2025 at 04:58 PM
Here's the first instance of breakage: If the admin supplies a v3 classic key and attempts to render a captcha checkbox, the box renders with an
error message, "ERROR for site owner: Invalid key type".
This might not happen in all cases. Within the Cloud/Enterprise console, it appears possible to create a challenge-type key. I need to try all of
those settings to figure out how to explain this to our users.miqrogroove - 7-6-2025 at 05:08 PM
So, two more things.
When creating a "Challenge" key through the Enterprise API, it returns a v2 type response.
When that same key is rendered as invisible, it renders with another error, "This site key is not enabled for the invisible captcha."
This is really confusing. I will stare at it for a while and hopefully see the larger pattern.miqrogroove - 7-6-2025 at 06:48 PM
Now running with a new reCAPTCHA key generated by Google Cloud with the checkbox challenge disabled. I'm happy with the results so far and
marked this feature resolved.miqrogroove - 7-31-2025 at 01:41 PM
That one is too slow and doesn't show reverse lookup names. I'd rather get rid of the link if there aren't any good ones.
Possible encoding error in that post. I need to review what happened there. Some of the later posts had apostrophes with no problem.miqrogroove - 7-31-2025 at 02:51 PM
I see in the commit history there was an inconsistency between the July 2 and July 3 changes. So that was simply a bug during the alpha and I will
edit the post to resolve that hiccup.
The reason it only showed up now is because the old post format used raw quotes and the introduction of encoded quotes was invisible until the
upgrade. The upgrade re-encoded all HTML, so any unexpected quote encoding would become double-encoded.
