kicad-developers team mailing list archive

[Jeff Young <jeff@xxxxxxxxx>]

More and more strange by the day.  My OSX build also shows that wxUSE_UNICODE_UTF8 is off (and thereby wxUSE_STRING_POS_CACHE).  Or at least CLion thinks it’s off (and highlights the code accordingly).

I had earlier encouraged Jon to flip the switch to see what happened.  I had meant locally, but perhaps he did it in master to see what he could smoke out?

Or perhaps someone else did it earlier?  Or perhaps the bug still exists even with wxUSE_STRING_POS_CACHE off? 


> On 3 May 2019, at 15:49, Jeff Young <jeff@xxxxxxxxx> wrote:
> 
> @Wayne, are you sure about those settings?  (That’s the flag I proposed we flip in the previous incantation of this thread.)  I’m fairly sure (although not 100% certain) that the bug only exists in wxUSE_UNICODE_UTF8 mode.
> 
> @Jon & @Tom, I know you guys have also fixed some of these crashes.  What platform did you find them on?  The ones I’ve fixed have been encountered on OSX (which is the only platform I have).
> 
>> On 3 May 2019, at 15:41, Wayne Stambaugh <stambaughw@xxxxxxxxx <mailto:stambaughw@xxxxxxxxx>> wrote:
>> 
>> On 5/3/19 5:22 AM, Jeff Young wrote:
>>> Hi Dick,
>>> 
>>>>> h) What is the list of deficiencies with current string usage?
>>> 
>>> I only have one issue with the current use of wxString, but it’s a big
>>> one: it crashes (unpredictably) when used multi-threaded in UTF8 mode.
>> 
>> I thought it was wxString itself that was not thread safe not
>> necessarily the utf-8 build but thread safety is the primary goal now
>> that we are using threads in multiple places within KiCad.
>> 
>> On my Debian system wx/setup.h shows
>> 
>> #define wxUSE_UNICODE 1
>> 
>> and
>> 
>> #define wxUSE_UNICODE_UTF8 0
>> 
>> so it would appear that wxString is built for unicode not utf8 mode on
>> linux.  I'm also pretty sure windows builds are unicode as well.
>> 
>> There is a secondary goal of removing wxWidgets from our low level
>> objects.  Maybe some day we can build the low level KiCad non-ui
>> libraries sans wxWdigets.  My thinking is that wxString should only come
>> into play at the UI level when dealing with wxWidgets UI code.  Being
>> able to use a standard C++ string implementation would (may?) go a long
>> way in helping with that goal.
>> 
>>> 
>>> This design document makes for fascinating
>>> reading: https://wiki.wxwidgets.org/Development:_UTF-8_Support <https://wiki.wxwidgets.org/Development:_UTF-8_Support>.  It
>>> appears that the current wxString is at least in part modelled on QtString.
>>> 
>>> There’s also a bunch of interesting info
>>> here: https://docs.wxwidgets.org/trunk/overview_string.html <https://docs.wxwidgets.org/trunk/overview_string.html>, which I
>>> believe is more up-to-date than the previous link.  In particular,
>>> there’s the mention that wxString handles extra-BMP characters
>>> transparently when compiled in UTF8 mode (currently used by Kicad), but
>>> does NOT when compiled in default mode (in which case the app must
>>> handle surrogate pairs).  This of course directly leads to your point (d):
>>> 
>>>>>> d) What does the set of characters that don't fall into UCS2
>>>>>> actually look like?  How big
>>>>>> is this set, really?  (UTF16 is bigger than UCS2 and picks up the
>>>>>> difference.)
>>> 
>>> Do we really need to handle extra-BMP characters?
>>> 
>>> An even more recent version of the second document
>>> (https://docs.wxwidgets.org/trunk/classwx_string.html <https://docs.wxwidgets.org/trunk/classwx_string.html>) finally makes an
>>> oblique reference to the multi-threading issue by starting with this
>>> (rather unhelpful) suggestion:
>>> 
>>> Note
>>>    While the use of wxString
>>>    <https://docs.wxwidgets.org/trunk/classwx_string.html <https://docs.wxwidgets.org/trunk/classwx_string.html>> is
>>>    unavoidable in wxWidgets program, you are encouraged to use the
>>>    standard string classes |std::string| or |std::wstring| in your
>>>    applications and convert them to and from wxString
>>>    <https://docs.wxwidgets.org/trunk/classwx_string.html <https://docs.wxwidgets.org/trunk/classwx_string.html>> only when
>>>    interacting with wxWidgets.
>>> 
>>> 
>>> Cheers,
>>> Jeff.
>>> 
>>> 
>>>> On 3 May 2019, at 02:03, Dick Hollenbeck <dick@xxxxxxxxxxx <mailto:dick@xxxxxxxxxxx>
>>>> <mailto:dick@xxxxxxxxxxx <mailto:dick@xxxxxxxxxxx>>> wrote:
>>>> 
>>>> On 5/2/19 5:32 PM, Dick Hollenbeck wrote:
>>>>> On 4/30/19 4:36 AM, Jeff Young wrote:
>>>>>> We had talked earlier about throwing the wxWidgets UTF8 compile
>>>>>> switch to get rid of our wxString re-entrancy problems.  However, I
>>>>>> noticed that the 6.0 work packages doc includes an item for
>>>>>> std::string-ization of the BOARD.  (While a lot more work, this is a
>>>>>> better solution because it also increases our gui-toolkit-choice
>>>>>> flexibility.)
>>>>>> 
>>>>>> I’d like to propose that we use std::wstring for that.  UTF8 should
>>>>>> *only* be an encoding format (similar to s-expr).  It should never
>>>>>> be used internally. That’s what unicode wchar_t’s are for.
>>>>>> 
>>>>>> And I’d like to propose that we extend std::wstring-ization to
>>>>>> SCH_ITEM and LIB_ITEM.  (Then we can get rid of a bunch of our ugly
>>>>>> mutex hacks.)
>>>>> 
>>>>> 
>>>>> I've been looking at this for a few months now.  I think it is so
>>>>> important, that a
>>>>> sub-committee should be formed, and if that committee takes as long
>>>>> as 4 months to come to
>>>>> a recommendation, this would not be too long.  This issue is simply
>>>>> too critical.
>>>>> 
>>>>> I would like to volunteer to be on that committee.  For the entire
>>>>> list to participate in
>>>>> this simply does not make sense to me.  I would welcome the
>>>>> opportunity to study this with
>>>>> a team of 5-6 players.  More than that probably leads to anxiety.
>>>>>  Then, given the
>>>>> recommendations, the list would of course have an opportunity to
>>>>> raise questions and take
>>>>> shots, before a strategy is formulated, and before anything is
>>>>> implemented.
>>>>> 
>>>>> Again, approximately:
>>>>> 
>>>>>  committee recommendations -> list approval -> strategy formulation
>>>>> -> implementation
>>>>> 
>>>>> 
>>>>> Up to now I have looked at many libraries and have [way *too* much]
>>>>> experience in multiple
>>>>> languages on multiple platforms, so I think I can be valuable
>>>>> contributor.
>>>>> 
>>>>> The final work product initially would simply be a list of
>>>>> recommendations, that quickly
>>>>> transforms to a strategy thereafter.  This is an enormous
>>>>> undertaking, so I suggest
>>>>> against racing to a solution.  It could look a lot easier than it
>>>>> will ultimately be, as
>>>>> is typical in software development.  But the return on investment
>>>>> needs to be near optimal
>>>>> in the end.
>>>>> 
>>>>> Some questions to answer are:
>>>>> 
>>>>> a) How did wxString get to its current state?  Is is merely a
>>>>> conglomeration of after
>>>>> thought, or is is anywhere near optimal.
>>>>> 
>>>>> b) Why so many forms of it?  Can one form be chosen for all platforms?
>>>>> 
>>>>> c) How does wxString it compare to QtString?
>>>>> 
>>>>> d) What does the set of characters that don't fall into UCS2 actually
>>>>> look like?  How big
>>>>> is this set, really?  (UTF16 is bigger than UCS2 and picks up the
>>>>> difference.)
>>>>> 
>>>>> e) For data files, I think UTF8 is fine.  So the change is for RAM
>>>>> manipulation of
>>>>> strings.  Aren't we talking about a RAM resident string that bridges
>>>>> into the GUI seamlessly?
>>>>> 
>>>>> f) What does new C++ language support offer?
>>>>> 
>>>>> g) What do C++ language designers suggest?
>>>> 
>>>> h) What is the list of deficiencies with current string usage?
>>>> 
>>>> 
>>>>> 
>>>>> 
>>>>> etc.
>>>>> 
>>>>> But this is best continued in a smaller group, as said.
>>>>> 
>>>>> 
>>>>> The other thing that I bring to this is vast familiarity with KiCad's
>>>>> internal workings,
>>>>> string use cases, and goals.
>>>>> 
>>>>> Let me know if I can help.
>>>>> 
>>>>> Regards,
>>>>> 
>>>>> Dick
>>>>> 
>>>>> 
>>>>> _______________________________________________
>>>>> Mailing list: https://launchpad.net/~kicad-developers <https://launchpad.net/~kicad-developers>
>>>>> Post to     : kicad-developers@xxxxxxxxxxxxxxxxxxx <mailto:kicad-developers@xxxxxxxxxxxxxxxxxxx>
>>>>> <mailto:kicad-developers@xxxxxxxxxxxxxxxxxxx <mailto:kicad-developers@xxxxxxxxxxxxxxxxxxx>>
>>>>> Unsubscribe : https://launchpad.net/~kicad-developers <https://launchpad.net/~kicad-developers>
>>>>> More help   : https://help.launchpad.net/ListHelp <https://help.launchpad.net/ListHelp>
>>>>> 
>>>> 
>>>> 
>>>> _______________________________________________
>>>> Mailing list: https://launchpad.net/~kicad-developers <https://launchpad.net/~kicad-developers>
>>>> Post to     : kicad-developers@xxxxxxxxxxxxxxxxxxx <mailto:kicad-developers@xxxxxxxxxxxxxxxxxxx>
>>>> <mailto:kicad-developers@xxxxxxxxxxxxxxxxxxx <mailto:kicad-developers@xxxxxxxxxxxxxxxxxxx>>
>>>> Unsubscribe : https://launchpad.net/~kicad-developers <https://launchpad.net/~kicad-developers>
>>>> More help   : https://help.launchpad.net/ListHelp <https://help.launchpad.net/ListHelp>
>>> 
>>> 
>>> _______________________________________________
>>> Mailing list: https://launchpad.net/~kicad-developers <https://launchpad.net/~kicad-developers>
>>> Post to     : kicad-developers@xxxxxxxxxxxxxxxxxxx <mailto:kicad-developers@xxxxxxxxxxxxxxxxxxx>
>>> Unsubscribe : https://launchpad.net/~kicad-developers <https://launchpad.net/~kicad-developers>
>>> More help   : https://help.launchpad.net/ListHelp <https://help.launchpad.net/ListHelp>
>>> 
>> 
>> _______________________________________________
>> Mailing list: https://launchpad.net/~kicad-developers <https://launchpad.net/~kicad-developers>
>> Post to     : kicad-developers@xxxxxxxxxxxxxxxxxxx <mailto:kicad-developers@xxxxxxxxxxxxxxxxxxx>
>> Unsubscribe : https://launchpad.net/~kicad-developers <https://launchpad.net/~kicad-developers>
>> More help   : https://help.launchpad.net/ListHelp <https://help.launchpad.net/ListHelp>
> _______________________________________________
> Mailing list: https://launchpad.net/~kicad-developers
> Post to     : kicad-developers@xxxxxxxxxxxxxxxxxxx
> Unsubscribe : https://launchpad.net/~kicad-developers
> More help   : https://help.launchpad.net/ListHelp