Bug 107952

To be more precise, fontconfig will select Kochi Gothic and Kochi Mincho, but
only if no other fonts that can provide the glyphs and claim to support language
"en" are available.  However, since the bitmap-fonts RPM must always be
installed for vte and hence for gnome-terminal, this means that the terrible
(for Japanese) MiscFixed font will be preferred for katakana and hiragana over
Kochi Gothic and Kochi Mincho.  (Similarly, Fangsong ti will be preferred, as
will the MS Japanese fonts if installed.)

That is, most applications will eventually fallback to Kochi Gothic and Kochi
Mincho, if no other fonts (including MiscFixed) are available.  Mozilla is even
worse-- it always completely refuses to use an "out of language" font,
regardless of glyph presence, and thus won't use Kochi Gothic and Kochi Mincho
for Unicode at all.

Maybe it is because Kochi Gothic's lang support does not listed en_US
as one of the supported locale. MiscFixed on the other hand, support
most of the langs (includes en_US). Is it the main reason, Owen?

That is why, yes.  There's been some discussion on the fontconfig list
about the right way to handle this.  The problem is that it's very
natural to want to have a preferred font used by the Sans alias for
viewing Japanese (and similarly for other languages), and there seems
to be no particular reason to insist that the font used for viewing
Japanese be able to handle English as a supported locale, if it's only
used for Japanese characters not provided in English fonts.

The problem is figuring out the right way for fontconfig to handle it.
 It's even worse when dealing with files in Unicode, which doesn't
have a language associated with the encoding.  Owen and Keith Packard
have discussed it some.

In the short run, adding broader coverage to Kochi Gothic and Kochi
Mincho would help.  In the long run, fontconfig's behavior is clearly
broken.  (It's unreasonble to expect that all fonts for all languages
contain all necessary Japanese glyphs so that fontconfig will be happy
when selecting them for users in Japanese locales, for example.)

pango-1.3.2, freetype-2.1.7 and other updates from rawhide/development
seem to go a long way towards resolving this problem.  I've also been
running fontconfig-2.2.92, but that didn't seem to really resolve it
until the pango and freetype updates.  In any case, closing as
RAWHIDE, since it looks resolved using development for FC2 right now.

Created attachment 104832 [details]
Screenshot of kana not using the same font as kanji

This is from Firefox PR10 and Fedora Core 3 Test2 (patched to current as of
Oct. 6, 2004)

This is a screenshot of google using English unicode with Japanese search
results. I increased the font size to emphasize how all the kana are not using
the same font as the kanji and how they don't scale or become anti-aliased.
This is NOT fixed and it has been happening since Fedora Core 1 (I run this at
work).

SJIS and ISO-2022-JP doesn't seem to have this problem. I can click into any of
these Japaense search results and they don't have the messed up kana like in
the screenshot.

Try putting the following in your /etc/fonts/local.conf file.  It will
cause fontconfig (at least on 2.2.96, but probably on the FC included
version as well) to match on Kochi Gothic and Kochi Mincho ignoring
language.  ("Strong" binding matches ignoring language.)

        <match target="pattern">
                <test name="family">
                        <string>sans-serif</string>
                </test>
                <edit name="family" binding="strong" mode="prepend">
                        <string>Luxi Sans</string>
                        <string>Bitstream Vera Sans</string>
                        <string>Verdana</string>
                        <string>Kochi Gothic</string>
                </edit>
        </match>
 
        <match target="pattern">
                <test name="family">
                        <string>serif</string>
                </test>
                <edit name="family" binding="strong" mode="prepend">
                        <string>Bitstream Vera Serif</string>
                        <string>Times New Roman</string>
                        <string>Kochi Mincho</string>
                </edit>
        </match>
                                                                     
          
        <match target="pattern">
                <test name="family">
                        <string>monospace</string>
                </test>
                <edit name="family" binding="strong" mode="prepend">
                        <string>Luxi Mono</string>
                        <string>Bitstream Vera Sans Mono</string>
                        <string>Andale Mono</string>
                        <string>Kochi Gothic</string>
                </edit>
        </match>

I tried that out. It works! Is this a hack or something that can be
commited into RAWHIDE CVS?

Created attachment 104882 [details]
After the change suggested to the local.conf file

Looks good and it scales correctly.

Well, it is a bit of a hack.

It's very much a "Western user who also uses Japanese"-centric
solution, and it also means that the language of one's locale (or the
language associated with a charset in Mozilla) will not be used to
choose the proper font.

It will result in the Japanese Kochi fonts always being preferred over
Chinese (and Korean) fonts for displaying Han characters.  For
Unicode, especially for English-based Unicode locales, there's no way
around this.  This results in Japanese fonts being used for
specifically Chinese webpages and Chinese charsets as well, from what
I remember.  This does create problems with the glyphs which have
different representations in the two languages.  Also, since
characters which are rare in Japanese and don't exist in the Kochi
fonts will be selected from the Chinese fonts, a somewhat inconsistent
character style sometimes results as well.

Also, this results in purely Japanese pages with Japanese charsets
using Luxi Sans (for Sans, and so on for Serif) for the
(non-doublewidth) Roman characters and Kochi fonts for other things,
rather than using Kochi fonts for everything when possible. 
Furthermore, regardless of one's locale, Luxi Sans will be used for
all characters it contains, rather than a language-specific font. 
When language-specific fonts have Latin characters designed to blend
in with their other glyphs better, this will mean slightly worse
looking characters.

Certainly not something that would go into the default fonts.conf.
It's  conceivable we could tighten up the set of required characters to
support 'en' if someone wanted to investigate what the Kochi
fonts are missing.

This problem will go away with the change of Mozilla to use
Pango, since Pango is smarter about selecting fonts. (It will
realize that an 'en' language makes no sense for Han/Kana
characters and subsitute it with 'ja' for Kana or 'xx' for
Han characters)

The required characters to support 'en' according to fontconfig which
are missing are all accented characters sometimes used (but sometimes
dropped) for foreign words which have been adopted by English.  They
are (thanks to fontconfig source and gucharmap):

0x00C0 (Capital Letter A with Grave)
0x00C7 (Capital Letter C with Cedilla)
0x00C8 (Capital Letter E with Grave)
0x00C9 (Capital Letter E with Acute)
0x00CA (Capital Letter E with Circumflex)
0x00CB (Capital Letter E with Diaeresis)
0x00CF (Capital Letter I with Diaeresis)
0x00D1 (Capital Letter N with Tilde)
0x00D4 (Capital Letter O with Circumflex)
0x00D6 (Capital Letter O with Diaeresis)
0x00E0 (Small Letter A with Grave)
0x00E7 (Small Letter C with Cedilla)
0x00E8 (Small Letter E with Grave)
0x00E9 (Small Letter E with Acute)
0x00EA (Small Letter E with Circumflex)
0x00EB (Small Letter E with Diaeresis)
0x00EF (Small Letter I with Diaeresis)
0x00F1 (Small Letter N with Tilde)
0x00F4 (Small Letter O with Circumflex)
0x00F6 (Small Letter O with Diaeresis)

If these characters could somehow be added to the Kochi fonts, the
fonts would support English according to fontconfig.

*** Bug 138783 has been marked as a duplicate of this bug. ***

I've hacked at fontconfig today to try and solve this bug.
What I did was to ammend the strict sort order used by FcFontSort so
that it 'satisfies' the language specified in the pattern by locating
the best matching font supporting each pattern language and then
ignores language in the remaining fonts for purposes of matching.

So, when asking for 'sans:lang=en', you'll get an English font first,
and then the remaining fonts sorted with respect to the 'sans' alias
alone -- pushing Kochi fonts ahead of other English-supporting Han fonts.

This change is in Fontconfig HEAD CVS; if someone would like to play
and see if it fixes the problems for them, that would be great.

I've been home for the holidays since the change was made, and have finally got
a chance to test it.  IMO, this solves all the problems that can be rationally
and consistently solved in a default configuration, and solves everything that
is fontconfig's problem.

Most remaining issues I can think of are clearly issues between pango, user
applications, documents, and individual configuration.  (For example, in an
English Unicode locale, view a UTF-8 document that isn't labeled for language
but is mostly Japanese.  Should the Latin characters be rendered with the
Japanese font used for everything else, or with the preferred English-supporting
font?  Similarly problems of preferring a Japanese vs. Chinese font for Han
characters in an unspecified document when in a non-CJK locale are up to
individual users.)

Now we just need a fontconfig release, and inclusion of that release into Fedora
Core and RHEL.  I know RH prefers to ship stable versions over development
versions, so even though I find the 2.2.9x series very stable, I'd be surprised
if RH included it.  What's preventing a 2.3.0 release?

I tried fontconfig-2.2.99 and it seems to fix bug 138783 too . :)

So, since 2.3.0 is released, I assume that Owen is working hard on editing the
various packages and default configuration, and we'll see it in RawHide soon,
right? ;)

Can we *please* update to 2.3.0 or 2.3.1 ( http://fontconfig.org/release/ ) in
time for FC4?  It fixes this (and some other important i18n and font rendering
problems).

Still appears to be fontconfig-2.2.3 in fc devel...

Owen told me he wanted to do this upgrade so I'm reassigning the bug to him.

If Sopwith does not accept the version upgrade (which is highly unlikely now
given that regressions in fontconfig will affect many people and we have little
time left before final), can the fix in 2.3.x be backported into our current
fontconfig?  Then that'll be our short-term solution.

Adding FC5Blocker for the long term solution, upgrade fontconfig.

The code portion is big enough that a backport, in my opinion, would be
riskier than a straight upgrade, and create significant deviation from
upstream, something that we want to avoid in Fedora.

Moving to FC5Target, as FC4 is 'done', barring release blockers that
pop up.

This bug could be changed to MODIFIED since we got fontconfig 2.3.2-1 on rawhide
now.

Light testing looks good to me.

Yes, this works for me too.  I'll close it as fixed in Rawhide.

*** Bug 107617 has been marked as a duplicate of this bug. ***