kicad-developers team mailing list archive

[Cirilo Bernardo <cirilo.bernardo@xxxxxxxxx>]

On Tue, Feb 28, 2017 at 1:07 AM, Wayne Stambaugh <stambaughw@xxxxxxxxx> wrote:
> On 2/26/2017 4:04 PM, Cirilo Bernardo wrote:
>> There is one other way which I found after much digging
>> and it involves a GCC extension. Since we use GCC on
>> Windows this might be acceptable:
>>
>> a. create a derived class of std::ifstream/ofstream. On
>> Windows the derived class will be used while on other
>> OS it will simply be typedef to std::ifstream/ofstream
>
> This seems reasonable to me.
>

 I had a look at the GCC STL implementation and unfortunately
is is impossible for me to implement (a) since I can't accomplish
what I want by deriving std::ifstream/ofstream due to the access
specifiers on the necessary member variables and the fact that
the open() function is not declared virtual.

>>
>> b. overload open() to use the gcc extension like this:
>> __gnu_cxx::stdio_filebuf buf( _wopen ( utf8_filename, _O_RDONLY ) );
>> std::istream mystream ( &buf );
>
> If this is portable, than I'm file with this as well but on the surface
> it looks gcc specific.  If that is the case, then I would rather got
> with option a.
>

Even solution (a), which I now know is not possible, would have been
a gcc-specific hack.

The solution I'm working on at the moment requires the replacement of

std::ifstream X;
X.open( filename, ... );
X.close();

with

OPEN_ISTREAM( X, filename );
CLOSE_STREAM( X );

On builds which are not MinGW the helper macros generate exactly
the same code as before. On MinGW builds, the helper macros
create an extra class which creates an i/ostream and cleans up
where required on destruction. The only caveat in MinGW is that
rather than an explicit ifstream/ofstream the object which is actually
created is an istream/ostream, but this is not a difficult thing to handle.
The good thing about preserving the use of std::iostream is that I
can eliminate some of the locale switching code and simply use
imbue() on the open streams to avoid unintended effects on other
code.

I still need to look into how the issue with OCE can be tackled then
see if the devs are willing to make changes. This UTF8 filename
problem has come up on the MinGW list many times and a number
of users had suggested various changes over the years but this
really seems to be a "won't fix" issue.

If we use anything other than gcc on Windows we can tackle the
other issues then.

- Cirilo

>>
>> The destructor must contain code to delete buf since
>> the istream will not delete it on destruction.
>>
>> If that would be acceptable I'll make some test
>> programs and work on a patch set. This solution
>> would also work on OCE and I can talk to the OCE
>> patch team to see what they think of it.
>>
>> - Cirilo
>>
>>
>> On Mon, Feb 27, 2017 at 4:54 AM, Wayne Stambaugh <stambaughw@xxxxxxxxx> wrote:
>>> On 2/26/2017 1:50 AM, Cirilo Bernardo wrote:
>>>> Hi folks,
>>>>
>>>>  This whole thing with UTF-8 filenames in Windows is a disaster.
>>>> What I've found so far:
>>>>
>>>> 1. Regarding OCE: Since OCE 0.17 (OpenCascade 6.8) UTF8
>>>> filenames have been supported when built with MSVC but
>>>> obviously not with MinGW.
>>>>
>>>> 2. MinGW does not provide any means of transparently using
>>>> UTF-8 filenames. All filenames within the STL *must* be
>>>> char* and MinGW *will* simply pass these on to OpenFileA()
>>>> on Windows resulting in UTF-8 being interpreted as ASCII-8
>>>> (and who uses ASCII-8 filenames anyway).
>>>>
>>>> So everything hinges on (2). If OCE uses std::stream then
>>>> fixing all issues under Windows is a lost cause. If OCE
>>>> simply plays with FILE* then it can be patched to work
>>>> in MinGW by invoking _wfopen() rather than fopen().
>>>>
>>>> As for kicad itself, std::stream is used in:
>>>> (a) VRML export
>>>> (b) IDF static library
>>>> (c) Scenegraph dynamic library for 3D plugins
>>>>
>>>> 2 paths forward come to mind and both will involve some work:
>>>>
>>>> (1) Move to the MSVC build system on Windows: this makes
>>>> it possible for us to use Microsoft extensions to STL to deal
>>>> with non-ASCII filename issues. There is no need to dig into
>>>> the OCE code since we know it will work correctly when built
>>>> with MSVC.
>>>
>>> This is not an acceptable solution.  It's not portable and would limit
>>> windows builds to using msvc.
>>>
>>>>
>>>> (2) Rework kicad code to play with FILE* (or wxFileStream)
>>>> rather than std::ifstream/ofstream. Although this will fix the
>>>> issues which are confined to kicad's source, it does nothing
>>>> to address the OCE issue. Whether or not OCE in MinGW
>>>> is a lost cause remains to be seen.
>>>
>>> FILE* is how we pretty much do it everywhere else in KiCad with good
>>> results so I don't see any reason not to do it this way with the model
>>> parser code.  At least it's portable across all build platforms.
>>> Doesn't oce have a reader function that takes a FILE *?
>>>
>>>>
>>>> One other possibility (but one which I hadn't looked into)
>>>> is to see if the STL implementation within MinGW uses the
>>>> MinGW-CRT. If it does then it may be possible to fix
>>>> everything by ensuring that the MinGW-CRT converts all
>>>> filenames to UTF16 and opens a file using FileOpenW().
>>>> In all cases this is not a pleasant task.
>>>>
>>>> Any comments/suggestions?
>>>>
>>>> - Cirilo
>>>>
>>>> _______________________________________________
>>>> Mailing list: https://launchpad.net/~kicad-developers
>>>> Post to     : kicad-developers@xxxxxxxxxxxxxxxxxxx
>>>> Unsubscribe : https://launchpad.net/~kicad-developers
>>>> More help   : https://help.launchpad.net/ListHelp
>>>>
>>>
>>>
>>> _______________________________________________
>>> Mailing list: https://launchpad.net/~kicad-developers
>>> Post to     : kicad-developers@xxxxxxxxxxxxxxxxxxx
>>> Unsubscribe : https://launchpad.net/~kicad-developers
>>> More help   : https://help.launchpad.net/ListHelp