[tw] What if?

[PMario]

What if ...

we would make commonmark [1] markdown [2] a 100% subset [3] of the
TiddlyWiki syntax?

-m

[1] http://commonmark.org/
[2] http://spec.commonmark.org/0.28
[3] https://en.wikipedia.org/wiki/Subset

[Riz]

Strongly supports the motion.

[Diego Mesa]

Strong second!

Post by Riz
Strongly supports the motion.

[@TiddlyTweeter]

Ciao PMario

I am very much of the view that TiddlyWiki could become a Universal Markup
handler (for the simpler markup systems).

Experimenting with BJ's http://flexibility.tiddlyspot.com/ ... a variant on
his Flexitypes that allows use of RegEx has made me very aware of a number
of issues ...

1 - IF you can access a PRE-PARSER then you can take any alien (simple)
Markup and convert it dynamically to TW markup (BUT only IF TW has an
equivalent). It is almost trivially easy. You do the conversion in the
pre-parse and then it passed through the normal rendering.

2 - That opens the possibility of using MANY types of naked markup,
including USER DESIGNED markup systems.

3 - COMPLEXITIES arise only when the originating markup has no TW direct
equivalent. For instance much of "Fountain Markup
<https://fountain.io/syntax>" does not have TW equivalents because its
IMPLICIT markup and TW mainly manages EXPLICIT markup.

Underlying my points here is a question about HOW much JavaScript you'd
actually need? A RegEx interface for pre-parsing seems to be able to do a
hell of lot (and without degrading performance).

Thoughts. With a bit experience in them.

Best wishes
Josiah

Post by PMario
What if ...
we would make commonmark [1] markdown [2] a 100% subset [3] of the
TiddlyWiki syntax?
-m
[1] http://commonmark.org/
[2] http://spec.commonmark.org/0.28
[3] https://en.wikipedia.org/wiki/Subset

[@TiddlyTweeter]

Further to my last.

I want to MOAN.

*Moan* that markup systems have emerged that differ but should be the same.

*Moan* that a universal etiquette on simple markup is not there. It
unnecessarily complicates things that the simpler markups are not
orthographically congruent.

*Moan* that TiddlyWiki isn't providing the needed Multi-Markup support is
not there yet.

*Hoping* that TiddlyWiki will take simple Markup and downs out of the dark
ages into a universal step forward.

Josiah, x

Post by PMario
What if ...
we would make commonmark [1] markdown [2] a 100% subset [3] of the
TiddlyWiki syntax?
-m
[1] http://commonmark.org/
[2] http://spec.commonmark.org/0.28
[3] https://en.wikipedia.org/wiki/Subset

[PMario]

Post by @TiddlyTweeter
I want to MOAN.

q:)

Post by @TiddlyTweeter
*Moan* that markup systems have emerged that differ but should be the same.

... We are lucky to have a choice! ... Otherwise everyone would have to use
what I prefer ;)

Post by @TiddlyTweeter
*Moan* that a universal etiquette on simple markup is not there. It
unnecessarily complicates things that the simpler markups are not
orthographically congruent.

Yea. ... Just "good enough" rules the world. You can see this everywhere.

*Moan* that TiddlyWiki isn't providing the needed Multi-Markup support is

Post by @TiddlyTweeter
not there yet.

Oh, ... we are close. ... BUT *if* we would make commonmark a first class
citicen, it would break backwards compatibility.

Post by @TiddlyTweeter
*Hoping* that TiddlyWiki will take simple Markup and downs out of the
dark ages into a universal step forward.

No hope here. If TW syntax would be a superset of MD it needs to be more
advanced and therefor more "complicated".

-m

[@TiddlyTweeter]

Josiah: TW isn't providing the needed Multi-Markup support is not there

yet.

Oh, ... we are close. ... BUT *if* we would make commonmark a first class
citicen, it would break backwards compatibility.

Initially, EEK! I looked at the 622 example cases of the CommonMark Spec.
<http://spec.commonmark.org/0.27/>But actually many of them are easy. If I
had a spare month I might have a stab at it (in regex) :-) Its the
multi-line nested blocks (like lists) that look most difficult.

Best wishes
Josiah

[Jeremy Ruston]

we would make commonmark [1] markdown [2] a 100% subset [3] of the TiddlyWiki syntax?

I’d be very interested if that were possible. It would potentially allow us to dispense with the current Markdown parser, which has significantly limited capabilities compared to native wikitext.

I also agree with Josiah’s point that a worthy longer term goal is to make TiddlyWiki a meta-markup system that allows end users to create their own markup parse rules. However, the complexity of the CommonMark specification suggests to me that it wouldn’t be possible to create a practical parser entirely declaratively. In other words, I think the complexities of arbitrary markup may well require arbitrary computational capabilities to parse.

But, of course, even if we couldn’t create a 100% spec compliant CommonMark parser declaratively, it would still be a very useful thing to have.

Best wishes

Jeremy

[@TiddlyTweeter]

Ciao Jeremy

I'm a pragmatist.

I simply do NOT believe that the 600 or so strictures that CommonMarkup
syntax say are essential are actually essential for most practical work. It
looks like a tall-story. When you look into it is boils down to something
more cope-able, I think.

I am pretty convinced that TW can get very close to a "universal markup"
for coping with most of the the variant Wiki-text versions in real use. And
I already know from playing with BJ's "pre-parser" that that there is
immense flex in what TW can do towards creating "User Defined Markup" quite
easily.

On this particular issue I'm happy to help as its one of the few things I
have competence on.

Best wishes
Josiah

we would make commonmark [1] markdown [2] a 100% subset [3] of the TiddlyWiki syntax?
I’d be very interested if that were possible. It would potentially allow
us to dispense with the current Markdown parser, which has significantly
limited capabilities compared to native wikitext.
I also agree with Josiah’s point that a worthy longer term goal is to make
TiddlyWiki a meta-markup system that allows end users to create their own
markup parse rules. However, the complexity of the CommonMark specification
suggests to me that it wouldn’t be possible to create a practical parser
entirely declaratively. In other words, I think the complexities of
arbitrary markup may well require arbitrary computational capabilities to
parse.
But, of course, even if we couldn’t create a 100% spec compliant
CommonMark parser declaratively, it would still be a very useful thing to
have.
Best wishes
Jeremy

[Jeremy Ruston]

Hi Josiah

Post by @TiddlyTweeter
I'm a pragmatist.
I simply do NOT believe that the 600 or so strictures that CommonMarkup syntax say are essential are actually essential for most practical work. It looks like a tall-story. When you look into it is boils down to something more cope-able, I think.

That’s rather my point: a putative declarative meta-markup couldn’t achieve 100% compatibility, but that even partial compatibility would be useful. But that’s not what Mario originally proposed.

Post by @TiddlyTweeter
I am pretty convinced that TW can get very close to a "universal markup" for coping with most of the the variant Wiki-text versions in real use. And I already know from playing with BJ's "pre-parser" that that there is immense flex in what TW can do towards creating "User Defined Markup" quite easily.
On this particular issue I'm happy to help as its one of the few things I have competence on.

Great, some concrete proposals would be useful at this point

Best wishes

Jeremy.

Post by @TiddlyTweeter
Best wishes
Josiah

we would make commonmark [1] markdown [2] a 100% subset [3] of the TiddlyWiki syntax?

I’d be very interested if that were possible. It would potentially allow us to dispense with the current Markdown parser, which has significantly limited capabilities compared to native wikitext.
I also agree with Josiah’s point that a worthy longer term goal is to make TiddlyWiki a meta-markup system that allows end users to create their own markup parse rules. However, the complexity of the CommonMark specification suggests to me that it wouldn’t be possible to create a practical parser entirely declaratively. In other words, I think the complexities of arbitrary markup may well require arbitrary computational capabilities to parse.
But, of course, even if we couldn’t create a 100% spec compliant CommonMark parser declaratively, it would still be a very useful thing to have.
Best wishes
Jeremy
--
You received this message because you are subscribed to the Google Groups "TiddlyWiki" group.
Visit this group at https://groups.google.com/group/tiddlywiki <https://groups.google.com/group/tiddlywiki>.
To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywiki/94321674-7dca-464a-abb2-aee48340beea%40googlegroups.com <https://groups.google.com/d/msgid/tiddlywiki/94321674-7dca-464a-abb2-aee48340beea%40googlegroups.com?utm_medium=email&utm_source=footer>.
For more options, visit https://groups.google.com/d/optout <https://groups.google.com/d/optout>.

[PMario]

Post by @TiddlyTweeter
I simply do NOT believe that the 600 or so strictures that CommonMarkup
syntax say are essential are actually essential for most practical work. It
looks like a tall-story. When you look into it is boils down to something
more cope-able, I think.
That’s rather my point: a putative declarative meta-markup couldn’t
achieve 100% compatibility, but that even partial compatibility would be
useful. But that’s not what Mario originally proposed.

I did have a closer look at the commonmark rules. ... They did standardize
some strange decisions. Mainly because of very popular existing MD software
and variants, such as pandoc ..

The new markdown spec allows the registration and usage of a variants hint
<https://tools.ietf.org/html/rfc7763#section-5>. So if we do it right, it
might be possible to use

text/markdown for 100% commonMark compatible stuff and
text/markdown; variant=tiddlywiki ... for our TW stuff. ... Only possibel
if we manage to register it!!


@Jeremy btw: We should register text/vnd.tiddlywiki to be listed by the
IANA. ...


We may be able to use: text/vnd.tiddlywiki; variant=commonmark or
variant=markdown-it or ...

So imo it's not really needed to get a 100% compatibility. .. As long as we
can manage the MD-zoo we would be able to create

have fun!
mario

[@TiddlyTweeter]

Post by PMario
So imo it's not really needed to get a 100% compatibility. .. As long as
we can manage the MD-zoo we would be able to create

Right. A likely ZOO. With likely cross-breeding over time. And demands to
allow some Darwin nightmare to live that should have been put-down.

Immediately for me come three potential tickets ...

TICKET #1 - The ONE-WAY TICKET - You join TW and it CONVERTS your markup to
TW markup. TW from now on, mate.

TICKET #2 - The TWO_WAY TICKET - You join TW and it CONVERTS your markup to
TW markup but lets you also EXPORT it to that terrible system you used
before. You are converted so now need to use the Catholic TW system but if
you want to go back to your previous bad habits you can.

TICKET #3 - The OPEN TICKET - You join TW and continue to use the markup
system you love already. Advantage: garnering the the lost souls of other
wiki systems. Disadvantage: the garnering of lost souls of other wiki
systems.

Just some thoughts.
Josiah

[TonyM]

Josiah,

Apart from a desire to use TiddlyWiki such that I can secure content from
any sources, markup alternatives etc.. I support multiple markups.

However I think a brainstorm could Identify specific uses for alternate
markups.


- I for one would love to be able to freely move content between
tiddlywiki and MediaWiki including Wikipedia such that I do not need to
know both in detail.
- I would like to be able to build scripts and batches, sql commands and
export them to their required file types which are basically plain text.
- I would like to define ad hoc markup to interrogate logs files and
other standard data sources.


I do however think HTML should be a 1st Class Citizen since regardless of a
markup standard, it is often displayed as html, almost every piece of
content can be captured and rendered this way.

Just some thoughts to test if your tickets are comprehensive enough.

Regards
Tony

[Tobias Beer]

I would think the solution is rather simple / has been done before:

I once implemented inline support for TWC using this type of syntax:

§§§
your markdown here
§§§

based on @ShowDown
<https://web.archive.org/web/20141021031548/http://showdown.tiddlyspace.com:80/>,
see docs at @MarkdownTutor
<https://web.archive.org/web/20151101000000*/markdowntutor.tiddlyspace.com>

It could easily be generalized into this:

§§§commonmark
your commonmark-markdown here
§§§

or even

§§§md-cm
your commonmark-markdown here
§§§

if we wanted it thusly.

best -tb

we would make commonmark [1] markdown [2] a 100% subset [3] of the TiddlyWiki syntax?
I’d be very interested if that were possible. It would potentially allow
us to dispense with the current Markdown parser, which has significantly
limited capabilities compared to native wikitext.
I also agree with Josiah’s point that a worthy longer term goal is to make
TiddlyWiki a meta-markup system that allows end users to create their own
markup parse rules. However, the complexity of the CommonMark specification
suggests to me that it wouldn’t be possible to create a practical parser
entirely declaratively. In other words, I think the complexities of
arbitrary markup may well require arbitrary computational capabilities to
parse.
But, of course, even if we couldn’t create a 100% spec compliant
CommonMark parser declaratively, it would still be a very useful thing to
have.
Best wishes
Jeremy

[@TiddlyTweeter]

Ciao PMario et al,

What if ... we would make commonmark [1] markdown [2] a 100% subset [3] of
the TiddlyWiki syntax?

Rather than CommonMark wouldn't Github Flavoured Markdown (GFM) be a better
aim? Its ...

1. compliant with CommonMark;
2. adds tables and other things Markdown does not do;
3. better services existing TW users who also use GitHub.

The specification comments ...

*GFM is a strict superset of CommonMark*. All the features which are

supported in GitHub user content and that are not specified on the original
CommonMark Spec are hence known as *extensions*, and highlighted as such.

See: GFM Specification <https://github.github.com/gfm/>

Early thoughts
Josiah

[PMario]

Post by @TiddlyTweeter
Rather than CommonMark wouldn't Github Flavoured Markdown (GFM) be a
better aim? Its ...

Nope. ... markdown is standardized now. (since ~ March 2016) ...
https://tools.ietf.org/html/rfc7763


Variants spec: https://tools.ietf.org/html/rfc7764#page-10

- GFM is one of several variants
- CommonMark is an other variant ...

Post by @TiddlyTweeter
1. compliant with CommonMark;
that's ok, but It's different
1. adds tables and other things Markdown does not do;
TW has tables that imo are more powerful
1. better services existing TW users who also use GitHub.
IMO no problem, we can have several variants. eg: markdown-it library

supports many variants already.
We just need to switch the mime type and we can deal with them. ...

I do have a finished plugin. ... I just have to prepare the edition.
... BUT ...
I want to get rid of external libraries, most of the time, they don't
integrate well with TWs possibilities.

have fun!
mario

[@TiddlyTweeter]

Which part of* Der Ring des Nibelungen *do you want to start with...? OK, Das
Rheingold

Post by PMario
I do have a finished plugin. ... I just have to prepare the edition.
... BUT ...
I want to get rid of external libraries, most of the time, they don't
integrate well with TWs possibilities.

First chorus: Do you have a full faultless CommonMark?

(expunges libraries)

Second chorus: No he doesn't.

(expunges libraries)

First chorus: He might have.

(expunges libraries)

Second chorus: Ask Him. Ask Him.

(expunges libraries)

First chorus: Your plugin is a full faultless CommonMark?

Notte, Josiah

[PMario]

Post by @TiddlyTweeter
First chorus: Your plugin is a full faultless CommonMark?

It uses the markdown-it library, which supports several MD variants. I did
include config options for commonmark, gfm, and markdown-it native.

-m

[@TiddlyTweeter]

Post by @TiddlyTweeter
First chorus: Your plugin is a full faultless CommonMark?
It uses the markdown-it library, which supports several MD variants. I did
include config options for commonmark, gfm, and markdown-it native.

That sounds brilliant. Especially with config options for variants.

Part of my concern is coming to TW form MD that its using its huge
flexibility to make migrants transfer to it easy.

Post by @TiddlyTweeter
Implementing MD as a first class citicen, for the basic writing syntax,
would imo lower the entry-level for new users quite a bit.

IMO, absolutely right. And enabling variants will make that aim much more
achievable. Because "MD" in the wild is in many forms, some with serious
extensions (GitHub & Dingus are the two I know of most).

I think practically that is the situation in the larger MD user bases--its
MD plus already... Offering options for variants could really help, I think.

Best wishes
Josiah

[TonyM]

Mario,

It is interesting you say "*TW has tables that imo are more powerful"* The
reason I say this is because only recently I came to discover exactly how
much of HTML is usable right in wikitext and tiddlywiki. Here is a working
example with a list widget in it.


<table style="width:100%; align: left; ">
<caption>Monthly savings</caption>
<tr>
<th>Item</th>
<th>Caption</th>
<th style="align: right;">Icon</th>
</tr>
<$list filter="[tags[Resources]]">
<tr>
<td>{{!!title}}</td>
<td>{{!!caption}}</td>
<td text-align: right;>{{!!icon}}</td>
</tr>
</$list>
<tr>
<th>Item</th>
<th>Caption</th>
<th >Icon</th>
</tr>
</table>

My Point is that html markup is often more powerful when dealing with more
sophisticated objects such as tables and simplifies the creation process
because elements are more easily duplicated and there is rudimentary error
checking and structure, CSS insertion points etc..

To me HTML is an essential skill and I see no harm choosing it when it
suits. With all the talk of alternate markup/down I hope we do not
compromise TiddlyWikis direct relationship to the pervasive standards on
the internet such as HTML/CSS especially since that is how the tiddlywiki
is rendered.

Theoretically would most markdown standards have documented processes to
markup up and down to/from HTML? If this was the case could we take any
HTML and apply any markdown to it?

My idea is we could produce any content that you can see on the screen,
Wiki Text, HTML, and any that works in Tiddlywiki, it is then rendered in
HTML and then generate markdown texts that can then be transferred into
their own tools.

An Example is I would Create a chapter in TiddlyWiki using any method I
choose, Display it (HTML) then choose to generate Wiki media markdown which
I then save, copy or paste into Wiki Media. Perhaps we could harvest
content in HTML from anywhere, Save it in Tiddlywiki and generate
alternative markdown from it, even to its orignnal mark down.

The all we have to do is translate any markdown to HTML and back to any
markdown. Which presumably is already done for most markdown formats.

Am I making sense?

Tony

Post by PMario

Post by @TiddlyTweeter
Rather than CommonMark wouldn't Github Flavoured Markdown (GFM) be a
better aim? Its ...

Nope. ... markdown is standardized now. (since ~ March 2016) ...
https://tools.ietf.org/html/rfc7763
Variants spec: https://tools.ietf.org/html/rfc7764#page-10
- GFM is one of several variants
- CommonMark is an other variant ...

Post by @TiddlyTweeter
1. compliant with CommonMark;
that's ok, but It's different
1. adds tables and other things Markdown does not do;
TW has tables that imo are more powerful
1. better services existing TW users who also use GitHub.
IMO no problem, we can have several variants. eg: markdown-it library

supports many variants already.
We just need to switch the mime type and we can deal with them. ...
I do have a finished plugin. ... I just have to prepare the edition.
... BUT ...
I want to get rid of external libraries, most of the time, they don't
integrate well with TWs possibilities.
have fun!
mario

[@TiddlyTweeter]

Practically WYSIWYIG is part of this.

The point of this is whether we still value (or need) the idea of PLAIN
TEXT representing something worth having. Markdown originally emerged for
email as much as for the web. The idea was to be able to AUTHOR in a text
style that was readable regardless of the final rendering.

I'm sceptical it applies now in the same way. Years have passed.

I DO think that SHORTCUT markup systems are still good. BUT. BUT. But like:
you can ALSO use shortcuts towards in HTML live editing too.

Its an interesting issue.

Whether TW is somewhat OVER-valuing "fading systems of mark-up" or not. I'm
personally divided on the issue but do see a tension.

The problem with HTML is only readability (the LOGIC is great, the visuals
of the actual code are crap). I'd far rather have my originating stuff in
TiddlyWiki / Markdown / MediaWiki syntax IF i don't have true WYSIWYG in
the editor. Once I have that I don't care.

Just thoughts
Josiah

... To me HTML is an essential skill and I see no harm choosing it when it
suits. With all the talk of alternate markup/down I hope we do not
compromise TiddlyWikis direct relationship to the pervasive standards on
the internet such as HTML/CSS especially since that is how the tiddlywiki
is rendered.

[TonyM]

Josiah,

Post by @TiddlyTweeter
The problem with HTML is only readability (the LOGIC is great, the visuals
of the actual code are crap). I'd far rather have my originating stuff in
TiddlyWiki / Markdown / MediaWiki syntax IF i don't have true WYSIWYG in
the editor. Once I have that I don't care

I understand your point, but when it is a complex, nested structure with
loops (lists) there is real value having open and closed elements, full
control of line breaks and classes/styles.

Sometime this even beats WYSIWIG because the structure logic and tag names
allow reference to the elements and logic there in.

I think there would be value identifying and documenting the use cases for
HTML usage in tiddlywiki.

Regards
Tony

[@TiddlyTweeter]

Personally I think a SWOP of what we currently have would be ideal. By
which I mean you edit in WYSIWYG and there is a pane that displays the
underlying code that (ideally you could also edit).

Likely much later. J, x

Post by @TiddlyTweeter
Josiah,

Post by @TiddlyTweeter
The problem with HTML is only readability (the LOGIC is great, the
visuals of the actual code are crap). I'd far rather have my originating
stuff in TiddlyWiki / Markdown / MediaWiki syntax IF i don't have true
WYSIWYG in the editor. Once I have that I don't care

I understand your point, but when it is a complex, nested structure with
loops (lists) there is real value having open and closed elements, full
control of line breaks and classes/styles.
Sometime this even beats WYSIWIG because the structure logic and tag names
allow reference to the elements and logic there in.
I think there would be value identifying and documenting the use cases for
HTML usage in tiddlywiki.
Regards
Tony

[TonyM]

I assume you have seen existing TiddlyWiki WISYWIG solutions?

Post by @TiddlyTweeter
Personally I think a SWOP of what we currently have would be ideal. By
which I mean you edit in WYSIWYG and there is a pane that displays the
underlying code (that ideally you could also edit).
Likely much later. J, x

Post by @TiddlyTweeter
Josiah,

Post by @TiddlyTweeter
The problem with HTML is only readability (the LOGIC is great, the
visuals of the actual code are crap). I'd far rather have my originating
stuff in TiddlyWiki / Markdown / MediaWiki syntax IF i don't have true
WYSIWYG in the editor. Once I have that I don't care

I understand your point, but when it is a complex, nested structure with
loops (lists) there is real value having open and closed elements, full
control of line breaks and classes/styles.
Sometime this even beats WYSIWIG because the structure logic and tag
names allow reference to the elements and logic there in.
I think there would be value identifying and documenting the use cases
for HTML usage in tiddlywiki.
Regards
Tony

[@TiddlyTweeter]

TonyM

I do use CKEDITOR in one TW. And I'm familiar with TinyMCE. Their expansion
to allow live editing of whole pages (can't do that in TW) is great. Their
growing model of "everything is editable" is a good one. A model where the
rendered version and the editable version are co-terminus.

J.

Post by TonyM
I assume you have seen existing TiddlyWiki WISYWIG solutions?

Post by @TiddlyTweeter
Personally I think a SWOP of what we currently have would be ideal. By
which I mean you edit in WYSIWYG and there is a pane that displays the
underlying code (that ideally you could also edit).
Likely much later. J, x

Post by @TiddlyTweeter
Josiah,

Post by @TiddlyTweeter
The problem with HTML is only readability (the LOGIC is great, the
visuals of the actual code are crap). I'd far rather have my originating
stuff in TiddlyWiki / Markdown / MediaWiki syntax IF i don't have true
WYSIWYG in the editor. Once I have that I don't care

I understand your point, but when it is a complex, nested structure with
loops (lists) there is real value having open and closed elements, full
control of line breaks and classes/styles.
Sometime this even beats WYSIWIG because the structure logic and tag
names allow reference to the elements and logic there in.
I think there would be value identifying and documenting the use cases
for HTML usage in tiddlywiki.
Regards
Tony

[PMario]

Post by @TiddlyTweeter
Practically WYSIWYIG is part of this.

*Different topic*. I want to discuss "What if " we include commonmark
Markdown into TW.

No discussion about WYSIWYG .. I personally thing it's broken, but that's
not the thread to discuss this.

-m

[@TiddlyTweeter]

Ciao PMario

My apologies if I messed up your thread :-(. Hopefully I didn't. It was not
intentional.

I will respond separately about why I think WYSIWYG is related in some ways
to this thread.

Best wishes
Josiah

Post by @TiddlyTweeter
Practically WYSIWYIG is part of this.
*Different topic*. I want to discuss "What if " we include commonmark
Markdown into TW.
No discussion about WYSIWYG .. I personally thing it's broken, but that's
not the thread to discuss this.

[PMario]

Post by @TiddlyTweeter
My apologies if I messed up your thread :-(. Hopefully I didn't. It was
not intentional.
I will respond separately about why I think WYSIWYG is related in some
ways to this thread.

WYSIWIYG is definitely related to any other text based markup system. ...
But it is a completely different philosophy. ... and discussions, between
those different point of views, can heat up, very quickly. ...

So I want to stay "text based" here. ... IMO prose mirror
<http://prosemirror.net/> could be a middle ground ;)

have fun!
mario

[Eric Shulman]

Post by PMario
WYSIWIYG is definitely related to any other text based markup system. ...
But it is a completely different philosophy. ... and discussions, between
those different point of views, can heat up, very quickly. ...

Back in the early days of font rendering (i.e., 1984, when the original
128K Macintosh with 9" monochrome screen was shipped), we had two other
acronyms:

WYSLPG = "Whistle Pig" (What You See Looks Pretty Good)

and the less friendly

WYGIWYD = "Wiggy Wid" (What You Get Is What You Deserve)

:)
-e

[PMario]

Post by TonyM
My Point is that html markup is often more powerful when dealing with more
sophisticated objects such as tables and simplifies the creation process
because elements are more easily duplicated and there is rudimentary error
checking and structure, CSS insertion points etc..

html is supported perfectly well. tables like your example is relatively
straight forward. ... but try to *dynamically nest* them, and you will feel
the pain.

With my - A script and todo manager workflow / UI experiment
<https://groups.google.com/forum/#!topic/tiddlywiki/_VQw8yIKWn8> video
series, I did explore dynamic table-like layouts, using CSS-grid
<https://caniuse.com/#search=grid>. I think a concept like this is much
easier to create and manage. .... With the experiment I did want to explore
CSS-grid, the possibilities it provides ... and I wanted to see, how much
"duplicated" code I produce to get it going. ... (Duplicated code let's you
find out patterns, that can be optimized ... in a second step)


To me HTML is an essential skill and I see no harm choosing it when it

Post by TonyM
suits.

The whole TW UI is build with dynamically created HTML. ... There is no
harm at all.

Post by TonyM
With all the talk of alternate markup/down I hope we do not compromise
TiddlyWikis direct relationship to the pervasive standards on the internet
such as HTML/CSS especially since that is how the tiddlywiki is rendered.

I don't intend to create the TW UI with markdown. ... TiddlyWiki as a
chameleon. That makes it powerful on one side and rases the complexity on
the other side.

Implementing MD as a first class citicen, for the basic writing syntax,
would imo lower the entry-level for new users quite a bit.

Theoretically would most markdown standards have documented processes to

Post by TonyM
markup up and down to/from HTML? If this was the case could we take any
HTML and apply any markdown to it?

I think there is a misunderstanding here.

Original markdown is a "wiki-syntax" for writing text
TiddlyWiki syntax is a "wiki-syntax" for writing text

Both systems take "plain text" and convert it to HTML, using predefined
rules, so browser can render it.

Those predefined rules define a: markup language

HTML means: .... HyperText Markup Language

eg: If we want to define something bold we use:

TW: ''bold text''
MD: **bold text**
HTML: <b>bold text</b>

The characters used to "mark" the enclosed text is 1 markup rule. ... "make
it bold" ... many such rules define a markup-language
<https://en.wikipedia.org/wiki/Markup_language> or markpu-syntax. There are
many markup-languages out there. TW, MD and HTML are only 3 of them.

TiddlyWiki and Markdown markup-syntax are designed to be human readable, *even
for big junks of prose text*.

HTML <https://en.wikipedia.org/wiki/Hypertext> was designed to display text
on a computer display to make it human readable. ... Since it is text it is
still readable by humans. But I don't want to read tables in HTML-text
format. You need it to be rendered into something, that humans can read.

Post by TonyM
My idea is we could produce any content that you can see on the screen,
Wiki Text, HTML, and any that works in Tiddlywiki, it is then rendered in
HTML and then generate markdown texts that can then be transferred into
their own tools.
An Example is I would Create a chapter in TiddlyWiki using any method I
choose, Display it (HTML) then choose to generate Wiki media markdown which
I then save, copy or paste into Wiki Media. Perhaps we could harvest
content in HTML from anywhere, Save it in Tiddlywiki and generate
alternative markdown from it, even to its orignnal mark down.

So what you want here is: *Convert one format to any other format*. That's
doable, but far away from simple.

With my example: **bold text** it's simple to convert from one language
into the other:

- Just replace MD ** with ''... and you have TW syntax .. super simple!
- HTML needs <b> for the beginning and </b> for the end. ...

So you can see, complexity goes up.

Now lets say we want red *bold text*. ... boom!! ... Markdown has NO rules
to define this, because MD primary goal is human readability!

TW and HTML have rules do deal with that.
...BUT
We have a fundamental problem now!

We will *lose information* if we want to convert <b class="red">bold
text</b> or ''@@.red bold text@@'' into Markdown. (I intentionally skip CSS
here)

The all we have to do is translate any markdown to HTML and back to any

Post by TonyM
markdown. Which presumably is already done for most markdown formats.

I hope you can see that it's not that easy. ... because your "roundtrip"
from MD to HTML and back to MD will fail for red *bold text*. ...

There are solutions to deal with that, but I think there is already too
much text :)

have fun!
mario

[PMario]

added video link to response.

Video 4++: discusses dynamic table-like layouts

http://youtu.be/3FcEHWnvtQU
-m
http://youtu.be/3FcEHWnvtQU

[PMario]

Post by TonyM
The all we have to do is translate any markdown to HTML and back to any

Post by TonyM
markdown. Which presumably is already done for most markdown formats.

Don't get me wrong. IMO every one of us would love to have this

functionality, and it should be one of our goals!

-m

[TonyM]

Mario,

Thanks for your comprehensive response. Sorry I took time to respond. I
was cognisant of most of these issues, I will review *dynamically nesting* and
see what the difficulty is.

Here is a response from me, such that after your effort, I do not stop the
conversation prematurely.

I understand the loss of information, however if a given markup does not
support CSS then the markdown from HTML will not include it (and must not).
The way to not loose information is to keep it, eg; retain the first
markdown text (in an additional field). The fact is with a nested list
driven by tiddlywiki the resulting HTML theoretically looses all the logic
applied to render it. I understand that.

I have used a "copy as HTML" Browser plugin in the past and pasted into the
code (HTML) window of wordpress, and the post looks as rendered in
tiddlywiki, the HTML render does not contain the widgets used to create it.
However you can see how I use tiddlywiki to generate sophisticated HTML and
using references to data in my wiki. I used to do the reverse with
<html>copy from browser selection</html> in TWC to some, but not total
success, This was to capture nicely formatted material off the internet.

In this above case the universal code is HTML, but of course I can't past
it into Wikipedia/WikiMedia (need to check that), but not as wikimedia
standard.

It seemed to me that if we chose HTML as a Go between markup, Then we made
tools to mark it down to one or more standards, for those that like a given
standard, then this could be reversed to generate HTML to display it in
tiddlywiki (as we must), but each new markdown need only have this To and
From HTML translation (presumable one way already exists) and you could
convert from any supported markdown to any other supported markdown, with
the inevitable losses if that standard does not support everything HTML can
render. HTML is the supper-set, the markdowns are a Subset. In a way this
will clean the markdown of unsupported features, in effect creating a
static and compliant version.

Of course TiddlyWIki may allow you to reintroduce css formats and even
widgets to the second markdown in your wiki, and do a full render back to
view it as html, but these will only work in tiddlywiki.

What would be interesting to see is if a reversible recode from HTML <>
markdown standard, was possible, just from the information needed to
describe "markdownstandard > HTML, after all this must be documented for
any markdown already.

It seemed like a more rapid and reusable way to handle alternative
markdowns.

But I must of course, defer to your greater experience of these things.

You have fun as well, I am :)

Regards
Tony

Post by PMario

Post by TonyM
My Point is that html markup is often more powerful when dealing with
more sophisticated objects such as tables and simplifies the creation
process because elements are more easily duplicated and there is
rudimentary error checking and structure, CSS insertion points etc..

html is supported perfectly well. tables like your example is relatively
straight forward. ... but try to *dynamically nest* them, and you will
feel the pain.
With my - A script and todo manager workflow / UI experiment
<https://groups.google.com/forum/#!topic/tiddlywiki/_VQw8yIKWn8> video
series, I did explore dynamic table-like layouts, using CSS-grid
<https://caniuse.com/#search=grid>. I think a concept like this is much
easier to create and manage. .... With the experiment I did want to explore
CSS-grid, the possibilities it provides ... and I wanted to see, how much
"duplicated" code I produce to get it going. ... (Duplicated code let's you
find out patterns, that can be optimized ... in a second step)
Video 4++: discusses dynamic table-like layouts
http://youtu.be/3FcEHWnvtQU
To me HTML is an essential skill and I see no harm choosing it when it

Post by TonyM
suits.

The whole TW UI is build with dynamically created HTML. ... There is no
harm at all.

Post by TonyM
With all the talk of alternate markup/down I hope we do not compromise
TiddlyWikis direct relationship to the pervasive standards on the internet
such as HTML/CSS especially since that is how the tiddlywiki is rendered.

I don't intend to create the TW UI with markdown. ... TiddlyWiki as a
chameleon. That makes it powerful on one side and rases the complexity on
the other side.
Implementing MD as a first class citicen, for the basic writing syntax,
would imo lower the entry-level for new users quite a bit.
Theoretically would most markdown standards have documented processes to

Post by TonyM
markup up and down to/from HTML? If this was the case could we take any
HTML and apply any markdown to it?

I think there is a misunderstanding here.
Original markdown is a "wiki-syntax" for writing text
TiddlyWiki syntax is a "wiki-syntax" for writing text
Both systems take "plain text" and convert it to HTML, using predefined
rules, so browser can render it.
Those predefined rules define a: markup language
HTML means: .... HyperText Markup Language
TW: ''bold text''
MD: **bold text**
HTML: <b>bold text</b>
The characters used to "mark" the enclosed text is 1 markup rule. ...
"make it bold" ... many such rules define a markup-language
<https://en.wikipedia.org/wiki/Markup_language> or markpu-syntax. There
are many markup-languages out there. TW, MD and HTML are only 3 of them.
TiddlyWiki and Markdown markup-syntax are designed to be human readable, *even
for big junks of prose text*.
HTML <https://en.wikipedia.org/wiki/Hypertext> was designed to display
text on a computer display to make it human readable. ... Since it is text
it is still readable by humans. But I don't want to read tables in
HTML-text format. You need it to be rendered into something, that humans
can read.

Post by TonyM
My idea is we could produce any content that you can see on the screen,
Wiki Text, HTML, and any that works in Tiddlywiki, it is then rendered in
HTML and then generate markdown texts that can then be transferred into
their own tools.
An Example is I would Create a chapter in TiddlyWiki using any method I
choose, Display it (HTML) then choose to generate Wiki media markdown which
I then save, copy or paste into Wiki Media. Perhaps we could harvest
content in HTML from anywhere, Save it in Tiddlywiki and generate
alternative markdown from it, even to its orignnal mark down.

So what you want here is: *Convert one format to any other format*.
That's doable, but far away from simple.
With my example: **bold text** it's simple to convert from one language
- Just replace MD ** with ''... and you have TW syntax .. super simple!
- HTML needs <b> for the beginning and </b> for the end. ...
So you can see, complexity goes up.
Now lets say we want red *bold text*. ... boom!! ... Markdown has NO
rules to define this, because MD primary goal is human readability!
TW and HTML have rules do deal with that.
...BUT
We have a fundamental problem now!
We will *lose information* if we want to convert <b class="red">bold
CSS here)
The all we have to do is translate any markdown to HTML and back to any

Post by TonyM
markdown. Which presumably is already done for most markdown formats.

I hope you can see that it's not that easy. ... because your "roundtrip"
from MD to HTML and back to MD will fail for red *bold text*. ...
There are solutions to deal with that, but I think there is already too
much text :)
have fun!
mario

[PMario]

Hi Tony,

Your Ideas are all right and sensible. ...

And we do want to have "roundtrips" to and from every possible content
format. ... The pandoc project is a very good example / implementaiton for
this workflow. .. With the exact same problems. (That's why they did their
own MD extensions..)

But

If we really want something reliable, *we can't allow data loss in the used
"storage format"!* ...

---------------- Some more internals and background

If you dig deeper into the topic you will inevitably end up at SGML
<https://en.wikipedia.org/wiki/Standard_Generalized_Markup_Language>
Standard Generalized Markup Language. SGML roots goes back to the 1960s ...
Simplified: it is a language / standard, to define other languages. To
achieve this, you need to increase complexity quite a bit. ..

SGML was originally designed to enable the sharing of machine-readable

large-project documents in government, law, and industry. Many such
documents must remain readable for several decades—a long time in the
information technology field
...source WikiPedia

You may ask: Why, not just implement SGML as our storage format and all
problems solved. .... IMO becasue our heads would explode. At least mine ;)
SGML is really hard and absolutely not human friendly by design. ... see
the quote from above ;)

That's why several "easier to handle" dialects have evolved: One of them is
XML... (still clunky and human unfriendly)

Once there was an attempt to force XML on the web with XHTML. ..But the
rules where so strict, that the browsers that implemented it broke the
existing web. .. The project blew up in front of their face ...

So we ended up with HTML5 .... Which is a standard that is "just good
enough", to be easy to use by machines and OK by humans.

----------------

As you found out HTML is a relatively good format to store "structured"
text. see: -> relatively. ... Because it was defined for a very specific
use-case. ... "The Web"

TiddlyWiki actually mis-uses it, to store tiddlers in HTML format. ... If
you open a tiddlywiki.hmtl file you can find elements like this. (scroll
down to line ~10.000++) <div author="JeremyRuston" core-version="&gt;=5.0.0" dependents=
"$:/themes/tiddlywiki/snowwhite" description="Centralises the story river"
name="Centralised" plugin-type="theme" title=
"$:/themes/tiddlywiki/centralised" type="application/json" version=
"5.1.16-prerelease">
<pre>{
html-escaped content there < is encoded as &lt; and > is &gt; ....
</pre>
</div>


This is the system tiddler: $:/themes/tiddlywiki/centralised were I
replaced the content with a 1 liner.

At TW startup this conent is transfered into something much more useful for
for javascript developers. -> JSON <https://en.wikipedia.org/wiki/JSON>(JavaScript
Object Notation)... It's easier to handle, because it is already part of
the js-language itself. No bulky 3rd party libraries are needed. JS itselfe
is just fine.

The cool thing with JSON is, it can also describe structues very well, in a
programmer friendly / readable format.

http://www.jsonml.org/ has some examples, where an html table is described
as JSON. ... Web programmers love JSON, because it is "just good enough" to
be extremely useful. Programs can handle json descriptions very well.

The purpose of JsonML is to provide a compact format for transporting XML-based

markup as JSON which allows it to be losslessly converted back to its
original form.
source jsonml.org

IMO the page is outdated, and there are probably newer projects, but the
idea behind it, is the right one.


TiddlyWiki uses JSON to convert

- tw-syntax -> into the parse-tree, the
- parse-tree is translated into -> the widget-tree
- widget-tree is translated into -> HTML output

If you open: https://tiddlywiki.com/prerelease/ and copy the following
text into the tiddler.


* line 1
* element 2


Open the preview panel

and select parse-tree;

Mode:block


[
{
"type": "element",
"tag": "ul",
"children": [
{
"type": "element",
"tag": "li",
"children": [
{
"type": "text",
"text": "line 1"
}
]
},
{
"type": "element",
"tag": "li",
"children": [
{
"type": "text",
"text": "element 2"
}
]
}
]
}
]

For this example the widget-tree looks similar, but if you enter
{{HelloThere}} you'll get a huge difference between parse and widget-tree.
Those *-tree like formats are relatively easy to handle by algorithms. ...



The problem with the TW parse-tree at the moment is, that it "looses
information". May be, because of performance reasons. Also implementation
speed is faster, and complexity is much less, if you skip functions.


So we actually can't use it, as a storage format. ...


BUT


If we would solve that problem, we could transform TW-syntax <-> Markdown
<-> HTML <-> TW-syntax <-> [you name it]


Such an itermediate storage fromat would be called an AST
<https://en.wikipedia.org/wiki/Abstract_syntax_tree> Abstract Syntax Tree.
While ASTs are known for programming languages, the mechanism we need is
the same. Our parse-tree and widget-tree basically are ASTs...


Some transformation paths exist and some _don't_. eg:


- tw-syntax -> AST -> HTML ... exists

- HTML -> AST -> tw-syntax ... doesn't exist


hope that helps


have fun!

mario

[PMario]

Hi,

I forgot to mention:

- tw-syntax -> AST -> HTML ... exists

Post by PMario
- HTML -> AST -> tw-syntax ... doesn't exist

If we use the AST as the tw-storage format, as a JSON file, we could use
this workflow

- AST -> HTML
- AST -> tw-syntax
- AST -> any syntax
- any-syntax -> AST

nice ... right?

-m

[@TiddlyTweeter]

PMario
... we could transform TW-syntax <-> Markdown <-> HTML <-> TW-syntax <->
[you name it]

No. Better.

Any Syntax -> Bits of that syntax TW supports/can translate already. ->
Then convert Unsupported Syntax -> HTML

Josiah

[@TiddlyTweeter]

PMario

The pandoc project is a very good example / implementaiton for this
workflow. .. With the exact same problems.

Right.

Yet encouraging *PANDOC to support TiddlyWiki format* would also be a
decent thing.

Just saying.

Josiah

[PMario]

Post by @TiddlyTweeter
Yet encouraging *PANDOC to support TiddlyWiki format* would also be a
decent thing.

The TW transclusion mechanism, is unique to TW. Most other system don't
even have an idea about mechanisms, that we take for granted.

-m

[@TiddlyTweeter]

It would be good for them to grasp transclusion better. Though some of the
Pandoc supported formats hold links to urls that are towards that.

In any case, I'm pointing at conversion FROM other formats TO TW to ease
migration.

TW's flex as already a enough of a super-set markup that its likely very
possible for Pandoc easy IMO. At least for the majority of simpler Wiki
markups it supports.

TW to other formats, I admit, may be much more difficult. But such limits
never stopped Pandoc before did it?

- j

Post by PMario

Post by @TiddlyTweeter
Yet encouraging *PANDOC to support TiddlyWiki format* would also be a
decent thing.

The TW transclusion mechanism, is unique to TW. Most other system don't
even have an idea about mechanisms, that we take for granted.
-m

[@TiddlyTweeter]

Ciao PMario & others interested in TW markup fexibility ...

Post by PMario
we would make commonmark [1] markdown [2] a 100% subset [3] of the
TiddlyWiki syntax?

The name "CommonMark" is slightly misleading (in implying that all Markdown
variants can be processed following its syntax. FWIW Markdown's maker,
Gruber, objected to their use of the word "Standard" in early versions so
they changed the name).

The initiative is a brilliant idea to create a superset of Markdown
variants but its actually *itself a variant* (a good variant, very
inclusive, less ambiguous, but NOT a total solution for all possible
cases).

The main breakage with some versions of Markdown is it ASSERTS a few rules
to reduce ambiguity in various implementations that are not fully backwards
compatible with all Markdown variants (of which there are many).

But I think its attractive because its getting used (both GitHub & Dingus
use it) in larger VOLUME sites.

*In looking at a possible TW way forward with it I think that the volume
usage cases are the most interesting & relevant*? And the actual way people
typically do mark-up in them, rather than the endless variants that
CommonMark posits as possible, could be a viable way?

Just early thoughts.

Best wishes
Josiah