How to enable multi-languages without strange characters appearing

[manny lee]

Prestashop support multi languages, but it can't support fully, because strange character appears in the front office in some languages.

I'm translating presta into Korean, and also got the strange characters in my front office, so searched all the forum in vain with no satisfiable answer. Followings are my solution.


I encoded translations into utf8 before writing into translation files, and decoded before actual use.
I just used two php functions utf8_encode() and utf8_decode().

That's all.


Below are changed files to get the correct utf8 character everywhere in prestashop.

your_admin_dir/AdminTranslations.php
classes/Pdf.php
classes/AdminTab.php
classes/Module.php
config/smarty.config.inc.php

The encoding part is writeTranslationFile function in AdminTranslations.php file.
Any other changed parts are about decoding.

Below is modified writeTranslationFile function in AdminTranslations.php.
All other changed parts are in my attatched files.

I commented with

// inserted by manny

and

/* original code */

at all the modified part.

   private function writeTranslationFile($type, $path, $mark = false, $fullmark = false)
   {
       global $currentIndex;

       if ($fd = fopen($path, 'w'))
       {
           unset($_POST['submitTranslations'.$type], $_POST['lang']);
           unset($_POST['token']);
           $toInsert = array();
           foreach($_POST as $key => $value)
               if (!empty($value))
                   //$toInsert[$key] = /*htmlentities(*/$value/*, ENT_COMPAT, 'UTF-8')*/; /* original code */ 
                   $toInsert[$key] = utf8_encode($value); // inserted by manny
           $tab = ($fullmark ? Tools::strtoupper($fullmark) : 'LANG').($mark ? Tools::strtoupper($mark) : '');
           fwrite($fd, "<?php\n\nglobal \$_".$tab.";\n\$_".$tab." = array();\n");
           foreach($toInsert as $key => $value)
               fwrite($fd, '$_'.$tab.'[\''.pSQL($key, true).'\'] = \''.pSQL($value, true).'\';'."\n");
           fwrite($fd, "\n?>");
           fclose($fd);
           Tools::redirectAdmin($currentIndex.'&conf=4&token;='.$this->token);
       }
       else
           die('Cannot write language file');
   }

You can use my attatched file and replace your original files with them.

My prestashop version 1.3 rc2.

Don't forget backing up your original files before any test!

Good luck!

AdminTab.php

AdminTranslations.php

Module.php

PDF.php

smarty.config.inc.php

[rocky]

You should report this on the bug tracker. If you are quick enough, it might get incorporated into v1.3 final.

[baudav]

PS is not full UTF8 compatible (PS use str, substr... must use mb_str;mb.. that is full UTF8 compatible)

but I search how to force mbstring overload; now if I set this in php.ini just BO is affected and not work. since 1.3 FO is not affected by mbstring overload and not found why!

I use IIS6 and php 5.3.2 with fastcgi.
I search if it necessary to set UTF8 output for IIS6 same as apache?

[rocky]

Prestashop already does use mb_substr if it is available. See the substr function in classes/Tools.php. Of course, if substr is called instead of Tools::substr, then the mb_substr function won't be used. There are quite a few places where Prestashop does this, but I'm not sure whether they do that for areas where UTF-8 isn't expected.

[baudav]

see bug 3983, 3871
bug 3871 just with IIS; PHP Warning: htmlentities(): Invalid multibyte sequence in argument in C:\DATA\WWW\tools\smarty\plugins\modifier.escape.php on line 29

corrected by convert before truncate; but now é is not 2 bytes, but more than 5 bytes (characters)
I have tested by mbstring overload, but no change.

[Matthieu Biart]

Hi everyone!

PS is not full UTF8 compatible

Prestashop already does use mb_substr if it is available. See the substr function in classes/Tools.php. Of course, if substr is called instead of Tools::substr, then the mb_substr function won't be used.

Rocky's right. PrestaShop is fully UTF-8 except for its PDF part (see specific PDF topic to learn more).
And like Rocky said, we trickly "overloaded" the PHP functions which do not support UTF-8.

but I search how to force mbstring overload; now if I set this in php.ini

So there is no need for such php.ini configuration. PrestaShop is capable to handle it by itself.

There are quite a few places where Prestashop does this, but I'm not sure whether they do that for areas where UTF-8 isn't expected

Exactly, concerned by performances we do not use the "overload" when not required (e.g., class names, iso codes...).


@manny lee: I replied to you on your bug report ;-)

[baudav]

OK, but I search to understand, why for me not work correctly.
exemple with bug 3871; resolved by truncate after convert (but not clean)
If I use truncate, and have a "à é â..." in the product name at position 35; Product name not displayed, and have this error:
PHP Warning: htmlentities(): Invalid multibyte sequence in argument in C:\DATA\WWW\tools\smarty\plugins\modifier.escape.php on line 29

after search, It's my truncate not work for UTF-8 and truncate a characters UTF8 in the middle.
I have test with one php file to test htmlentities

$str = 'câble TOSLink mâle / mâle (1.5 à 25m)';
$etc = '...';
echo htmlentities(substr($str, 0, 35). $etc, ENT_QUOTES, 'UTF-8');
echo htmlentities(mb_substr($str, 0, 35). $etc, ENT_QUOTES, 'UTF-8');
echo htmlentities(mb_substr($str, 0, 35,'UTF-8'). $etc, ENT_QUOTES, 'UTF-8');

first not work and report same error; but work if I overload and set utf8 in php.ini.
second work if I set UTF8 in php.ini
third work allways.
My PHP work correctly, error is normal ?

I suppose It's UTF8 problem; but now with your response, I'm not sure. (for PS 1.2.5, overload resolve problem, but now with SVN overload no change, just BO not work if I enable overload)
Can you help me to find my config error ?
I use PHP 5.2.3 vc9 nts with FastCGI1.5

which optimal parameter for mbstring set to php.ini ?

@Manny Lee: problem with strange characters is in FO and BO ? can you post screenshots ?
can you share your korean language pack ?
I use simplified-chinese without problem.
Perhaps you not use UTF-8 in your translation file? (if you use Windows, test with notepad++: unicode,utf8,ascii... editor with syntax color... very good, and easy to convert file to UTF8)

I not understand why use utf8_encode() and utf8_decode() if all is UTF8.
but found recently, to avoid error or no display, you can use iconv('utf-8','utf-8',$str); (my next test for my bug with utf8 product name: htmlentities(iconv('UTF-8', 'UTF-8//IGNORE', $str), ENT_QUOTES, 'UTF-8');

[manny lee]

Hello, everybody.

Here are two page URLs where you can compare the bug.

http://nshop.selfip.net/product.php?id_product=13
http://presta.selfip.net/product.php?id_product=7

Select Korean and check the “Availability” message on each page. You can’t miss it.

Both of sites are on my test server. One is original presta, and the other is pateched with my solution.

I inputted “이 모델에 해당하는 상품이 없습니다. 다른 것을 선택해 주십시오.” at the plaec of “The product does not exist in this model. Please choose another.” on Product Translations at the Translations tool in BO.

I'am using Linux, php5, apache2, mysql5. Their default character set is utf8.

I attatched two screenshots.

I got the strange character on FO, but not on BO. But, I had to modify BO files to see the correct characters on both sides.

[baudav]

no problem for me. (IIS6, php 5.3.2)

You have lastest PHP ? 5.3.2

see php.ini (but I think this value is forced by PS)
default_charset = "utf-8"
[mbstring]
mbstring.language = UTF-8
mbstring.internal_encoding = UTF-8
mbstring.http_input = UTF-8
mbstring.http_output = UTF-8
mbstring.encoding_translation = Off
mbstring.detect_order = auto
mbstring.substitute_character = none
mbstring.func_overload = 0


and apache conf: (httpd.conf and other)
I use apache before, and nothing for language set in config. (but you can see AddDefaultCharset UTF-8)


see file /theme/prestashop/lang/kr.php (2 letters for language)
search this line, and see if your file is correctly saved in UTF8 format (for notepad++ is displayed as "ANSI as UTF-8"):
$_LANG['product_e4aae275fc6b4d57daab4431b8bf736c'] = '이 모델에 해당하는 상품이 없습니다. 다른 것을 선택해 주십시오.';
if not, test with edit directly file and save in UTF8.

[manny lee]

misou.cn,

There was just one left I didn't adapt, it was mbstring section in php.
So, I reconfigured my php.ini as your suggestion.

Alas... the result is the same.

Yes, I confirmed the language file already. I could not see the correct characters in the ko.php file. but I could see the right charcters in BO. I thought that there must had been an error while writing the language file. It was my intention to prevent the writing error. So, I used the utf8_encode() and utf8_decode().

The solution works well in my case.

Have you tested on linux system? My environment is as below.

Linux kernel 2.6.18, Apache 2.0.63, PHP 5.2.6, MySQL 5.0.67

Thanks.

[baudav]

but it's not normal use encode and decode if all is utf8.
perhaps korean contain characters with more 2 bytes when you type in BO and your encore/decode reencode this in UTF8 (same as iconv(‘utf-8’,‘utf-8’,$str); )

can you test: you copy this that is UTF8 "이 모델에 해당하는 상품이 없습니다. 다른 것을 선택해 주십시오." and paste in BO with english keyboard.
I not know how you type korean, but I think is same as chinese.

and test, if you edit and paste this directly in your ko.php; if FO can display correctly.


At first I thought that our problems could have the same cause, but for me, I'm heading for the rewrite functions that do not work.

[manny lee]

Sorry misou.cn.

That still does not work.

[baudav]

Hi manny lee,
Do you have a platform to test locally?
you can test with latest php/apache and see if not a bug with this ?

I use VMWare server (it's free), perhaps you can test, and found virtual disk preintalled with Linux,apache,php.

My problem is also related to UTF8, and like you, I am the only one having this problem with a temporary solution that works. soon as I get time I return to the problem. If it works for all others, is necessarily a problem of compatibility or a config error.

[baudav]

I think your problem is: korean use 3 bytes by characters and your space only one (same as chinese)
test with chinese: 该货品不存在。请选择其他的货品。(3 bytes /chars)
test with chinese and space: 该 货品 不存在。请选 择其他 的货品
and test with éà üÉ ôâ ûê (2bytes/chars)