XMB Forum Software

[Resolved] Critical error: Usage of norwegian special characters (æ, ø, å) blanks the post.

solbu - 8-5-2013 at 10:53 PM

Full Board URL: http://forum.solbu.net/
XMB Version: 1.9.11

I run a norwegian forum. Whenever someone posts anything containing the norwegian special characters «æ», «ø» and «å», the entire post is totally BLANK both on preview and when posting.

The apache logs give no errors.

Anyone have any idea as to what is going on?
(The forum is running on Debian 7)

lottos - 8-6-2013 at 12:10 AM

Which character set are you using on the database - see http://forums.xmbforum2.com/viewthread.php?tid=776873

Has this just started to occur?
Was there an upgrade to mysql or php on your server?

eman86 - 8-6-2013 at 12:20 PM

I think if the tables were set to UTF-8 it might resolve that issue for you.

solbu - 8-7-2013 at 08:10 AM

Quote: Originally posted by lottos  
Which character set are you using on the database - see http://forums.xmbforum2.com/viewthread.php?tid=776873
It seems to be latin1_swedish_ci.

Quote:
Has this just started to occur?
Was there an upgrade to mysql or php on your server?
It happened sometimes this summer. I don't know exactly when, as the forum is not a wery active one, and I don't use it each week myself.
But we did upgrade from Debian 6 to Debian 7 this summer, so there most definetly was an upgrade of both MySQL and php.

Quote: Originally posted by eman86  
I think if the tables were set to UTF-8 it might resolve that issue for you.
Interesting. And how do we do that? :)

eman86 - 8-7-2013 at 01:03 PM

you have go and change that manually on each xmb table. give some some time and I can write up a nifty script for you :)

solbu - 8-8-2013 at 07:27 AM

Quote: Originally posted by eman86  
you have go and change that manually on each xmb table. give some some time and I can write up a nifty script for you :)
Hmm. I found something that did that, and the special characters still blanks the posts. And changing it somehow broke every configured setting throughout the entire forum. Like Sub forum names, the rules, etc.. So we had to revert all changes by importing from backup.

lottos - 8-8-2013 at 09:05 AM

Quote: Originally posted by solbu  

It happened sometimes this summer. I don't know exactly when, as the forum is not a wery active one, and I don't use it each week myself.
But we did upgrade from Debian 6 to Debian 7 this summer, so there most definetly was an upgrade of both MySQL and php.


Perhaps you can find out what versions you were running and what versions you upgraded to.

Mouser - 8-8-2013 at 03:35 PM

Always good to make a back-up if you do changes like that.

However, changing the charset should not break all configured settings.

Could you please show us a screenshot of what you changed?

solbu - 8-16-2013 at 04:32 PM

First, sorry for late response.

Quote: Originally posted by Mouser  
Always good to make a back-up if you do changes like that.
No problems there. We have an automated (crontab) daily database backup routine that keep backups for 5 months back.
Quote:
Could you please show us a screenshot of what you changed?
We reverted to the old nonworking backup, so all is the same as it where when the operating system was upgraded, so I don't have a screenshot. I also remember the php script I used (and ran in the terminal) had problems converting some of the tables and exited, resulting in me having to run all of the steps manually anyway, and stil had some tables that would not convert, so I can't use it again.

Any other ideas?

solbu - 8-16-2013 at 04:32 PM

Quote: Originally posted by eman86  
you have go and change that manually on each xmb table. give some some time and I can write up a nifty script for you :)
I think I'll have take you up on the offer. :)

eman86 - 8-16-2013 at 08:29 PM

I'll try to whip something up this weekend and will post back when I'm done :)

eman86 - 8-21-2013 at 12:50 AM

here it is, this will convert every xmb table to UTF-8

Attachment: xmb-sql-utf8.php (3kB)
This file has been downloaded 690 times

Adalberto - 8-21-2013 at 11:47 AM

Quote: Originally posted by eman86  
here it is, this will convert every xmb table to UTF-8

Thanks ;)

Mouser - 8-21-2013 at 08:31 PM

On my forum, the charset is "latin1_swedish_ci" and have not had any issues so far. *crosses fingers*


The provided PHP file, works if the table_prefix is "xmb_".
Q; Why not include the config.php and pull the database info and table_prefix from there? ;)




7 Years ago I made some handy scripts for versions like XMB1.9.6.
I've taken one of those files and modified the queries to be run.
This will work without the need to modify it, just upload it to the forum-root and run it... Then click on the link to start the procedure to update your tables to charset "utf8_unicode_ci".


Please, create a backup of your database before you start this procedure !!

Attachment: TablesEncoding.php (5kB)
This file has been downloaded 529 times

eman86 - 8-22-2013 at 12:14 PM

I was going to do that mouser but since XMB's codebase is a bit new to me I wasn't sure if the config had anything sort of "anti-theif" check to it. 99.99% of people don't change the prefix imo at from my experience most don't do it unless theres a real reason to.

Adalberto - 8-22-2013 at 07:50 PM

Hi, problem in my forum is when one member write message with this chars:
à è ì ù à ò

If I use php 5.3 or lower work correctly, but I see blank window with php 5.4 or higher. Only solution is change charset with utf8_unicode_ci :)

solbu - 9-8-2013 at 06:45 AM

I've now tested both scripts, and nothing changes. As in, the error of blank posts still remains.

lottos - 9-8-2013 at 08:35 AM

Seems to be a known issue with php 5.4:

http://stackoverflow.com/questions/15028339/odd-issue-with-f...


This implies reloading the data into the database fixed it:

http://stackoverflow.com/questions/14049079/php-5-4-9-mysql-...


vbulletin also had issues:

http://www.vbulletin.com/forum/forum/vbulletin-4/vbulletin-4...

with this the possible fix, if implemented into xmb:

http://www.vbulletin.com/forum/forum/vbulletin-4/vbulletin-4...

http://www.vbulletin.com/forum/forum/vbulletin-4/vbulletin-4...


Along with SMF, with this their possible fix:

http://www.simplemachines.org/community/index.php?topic=4971...


and Wordpress:

http://core.trac.wordpress.org/ticket/24121


Another possible workaround is step 4 from

http://xaviesteve.com/1223/issues-with-accents-and-strange-c...



I suspect the culprits are the post.php file:

$rawsubject = htmlspecialchars_decode($threadname, ENT_QUOTES);
$rawusername = htmlspecialchars_decode($username, ENT_QUOTES);
$rawemail = htmlspecialchars_decode($subs['email'], ENT_QUOTES);
$rawbbname = htmlspecialchars_decode($bbname, ENT_NOQUOTES);

Example:
$rawusername = htmlspecialchars_decode($username, ENT_QUOTES);
probably needs change to:
$rawusername = htmlspecialchars_decode($username, ENT_QUOTES, 'utf8');

In functions.inc.php,
function htmlspecialchars_decode($string, $type=ENT_QUOTES)
probably also needs changing.

If that works, there are another six instances in u2u.inc.php, possibly others in other script files.


Although... probably best to have the settings or config.php have the character set type rather than hard code.

solbu - 9-8-2013 at 11:45 AM

Quote: Originally posted by lottos  
probably best to have the settings or config.php have the character set type rather than hard code.
Then the question becomes, how do one do that. :)

Daniel Gouveia - 9-9-2013 at 02:49 AM

I'm a bit far from the xmb but I'll try to help.

The problem is the php 5.4 like lottos said.

Maybe this can fix this problem but is not a permanent solution

Just replace the file validate.inc.php

Attachment: validate.inc.php (14kB)
This file has been downloaded 621 times

solbu - 9-12-2013 at 08:51 PM

Quote: Originally posted by Daniel Gouveia  
The problem is the php 5.4 like lottos said.

Maybe this can fix this problem but is not a permanent solution

Just replace the file validate.inc.php

And that did the trick. Thank you so much.

SFSW - 5-26-2014 at 07:57 AM

Thank you for this fix, I ran into this same php 5.4 problem recently. The new validate.inc file generally works, however, the new validate file apparently disables changing the boards settings in the admin control panel. Likely due to changes in the array items check section of the file to include the $charset option. Would there be a fix available for this? I've been trying to change things around myself, but am concerned I'll break some other functionality.

Edit: To expand on this, I activated the debug flag and the problem occurs at the 'bbrulestxt' part of the 'UPDATE xmb_settings' query. The error that occurs is:

(errno = 1064)

Apparently some kind of problem with ' characters/quotes being used within the text content of the array. Still hunting for a solution...

SFSW - 5-27-2014 at 02:17 AM

I've managed to pin down the cause and come up with a work-around, but unfortunately, fixing the initial problem breaks other functionality within the validate function. Here are the details, maybe someone else could provide a better solution with this information:

Magic quotes has been removed in php 5.4, so the get_magic_quotes_gpc() call will always return false. Bypassing that check is simple:

// if (get_magic_quotes_gpc()) {
$retval = stripslashes($retval);
// }

That removes the magic quotes check and runs the stripslashes function to get rid of the ' delimiter before and after the string. Here is where things get tricky and start to break down. Due to something in the new validate.inc file, any delimiters within strings must be marked with a slash, otherwise they will divide the strings in the wrong places. This is what causes the errorno 1064 with the new validate.inc file. In order for the admin settings option to work, the delimeters must be marked with slashes, so the above can be modified like this:

// if (get_magic_quotes_gpc()) {
$retval = stripslashes($retval);
$retval = addslashes($retval);
// }

This effectively adds slashes to the delimiters left behind (namely any apostrophes), which allows the admin settings to function properly. However, because of this change, regular forum posts will now have slashes ahead of any apostrophes, altering the post content in an undesired way. So there just needs to be a way to allow one to include the slashes while the other doesn't or maybe some other solution. Anyone have ideas for an effective solution to this?

Ideally, any apostrophes in strings should just not be treated as delimiters. The old system worked this way, but I'm having trouble coming up with the solution for the new validate.inc file under php 5.4.

SFSW - 5-28-2014 at 05:22 AM

Here is more information and a solution to the problem I mentioned above.

If you rem out the old magic quotes part of the validate.inc file entirely and add 'addslashes' to the requires fields in the settings section of the cp.php file, the settings option will then work with this new validate.inc file that works with php 5.4. Here are the steps:


In validate.inc.php, find:

if (get_magic_quotes_gpc()) {
$retval = stripslashes($retval);
}

rem these lines out so they look like this (magic quotes returns false in php 5.4 anyway):

// if (get_magic_quotes_gpc()) {
// $retval = stripslashes($retval);
// }


Then in cp.php, find:

$bbrulestxtnew = postedVar('bbrulestxtnew', '', FALSE);

add the 'addslashes' command like this:

$bbrulestxtnew = addslashes(postedVar('bbrulestxtnew', '', FALSE));


Then in cp.php, find:

$bboffreasonnew = postedVar('bboffreasonnew');

add the 'addslashes' command like this:

$bboffreasonnew = addslashes(postedVar('bboffreasonnew'));


Then in cp.php, find:

$tickercontentsnew = postedVar('tickercontentsnew');

add the 'addslashes' command like this:

$tickercontentsnew = addslashes(postedVar('tickercontentsnew'));


With these changes and additions in place, the settings in the admin control panel will work correctly even if a delimeter is used in any of these three text fields (otherwise changes to the settings will fail without these changes). There may be other parts of the codebase that need this kind of modification in order to work.

So please, if someone else who is familiar with the XMB codebase knows of any other entry areas that need to have slashes added to prevent functions from failing, please post any other required changes to keep XMB boards working properly on PHP 5.4.

If I find any others, I'll try to post them here, but I'll likely only find them through trial and error as I try to use the board over time.