Key Pages
Home |Stage 1 - consolidate |
Stage 2 - moving on |
The big picture |
Single page |
Table of Contents |
Changes
Key Pages
Home |
NexTcl + NexTk - a possible roadmap for Tcl/Tk development
(a.k.a. The NOLA Roadmap)
Management Summary
Tcl/Tk 8.5 is the best yet - with lots of new features. But the perception is that Tcl is moribund and Tk is dated, and there are barriers to new users (such as too much choice and no consensus as to the set of useful extensions).
In addition, whilst the TCT and TIP process can deliver stability and reliability, to date it hasn’t delivered the rapid progress expected in the “Internet age” (as evidenced by the 5 years it took to produce Tcl/Tk 8.5).
This roadmap outlines a strategy to address these issues, through
Contents
Based on many discussions at Tcl2007 in New Orleans (hence the "NOLA Roadmap"), this “roadmap” is akin to a scoping study for a commercial product plan. It’s a start, and needs to be fleshed out, but in the absence of any other consolidated roadmap it is presented as a way forward that attempts to recognize the issues and limitations facing the Tcl community, and to reconcile the various viewpoints put forward.
Key goals
As Kevin pointed out, this is all “mom and apple pie” - but consider for a moment how the current TCT/TIP process rates against these goals? The opinion of many at the conference is that it doesn’t rate too well (notwithstanding the significant efforts of the TCT and others to get 8.5 to beta stage).
The TCT and TIP process can deliver stability and reliability, but to date it hasn’t delivered the rapid progress expected in the “Internet age” (as evidenced by the 5 years it took to produce Tcl/Tk 8.5). We need to find ways of doing both.
Strategies
Tcl/Tk 8.6 would be Tcl/Tk 8.5 plus a key new feature - TclOO.
Why TclOO in Tcl 8.6? For a number of reasons:
In the case of Tcl, classic means “maintained but not actively developed” - that is, from 8.5 onwards Tcl is only enhanced with features needed to improve Tk, and any development effort is focussed on the next generation of Tcl (see later).
In the case of Tk, classic means “I’d want that on my bookshelf” - i.e. there may be a series of releases (Tcl/Tk 8.6, 8.7, etc) but they are focussed on achieving a modern look and feel out of the box, with new defaults and new widgets (and/or integration of third party widgets)
What goes into Tk 8.6?
Basically, anything that will make Tk 8.6 look good out of the box and is relatively low hanging fruit. It doesn’t have to be perfect, just sufficient - especially given the New Tcl and New Tk efforts might radically change the language implementation.
An 8.6 classic release would also be a basis for people seeking to develop a reasonably sized distribution containing best of breed or common extensions (ref. Larry’s ideas) - see modular Tcl below.
Notes
Amongst the developers that we are trying to reach are those who are looking for a GUI toolkit and who would be tempted by Python, Ruby, or Perl or (to a lesser degree) VB if they didn't know any better. And to people already using those languages who need a GUI toolkit.
The positioning in the shorter term is to modernize and relaunch the "Tk story". That is, scripting lets you very cleanly and concisely put together a GUI that's as good as native, with way less effort (we aren't there now, because it can't be native without a lot of picky work, so you have "native" or "concise", not both). When used appropriately Tk is arguably better than the alternatives (Cocoa, wX, GTK/QT, Java/SWT, VB), and will be the “gentle” introduction to Tcl for people in other language communities.
The goal of modernizing and relaunching the Tk story is to reclaim the GUI high ground.
How can this be done? Jeff’s suggestion of updating the Tk bindings to other languages is worth expanding on. This, combined with the better “out of box experience” would be the start of clawing back Tk’s reputation amongst other language communities (Jeff - expand please).
The Tcl browser plugin probably needs to be part of the story too, providing finishing doesn’t divert resources from integrating with other languages. The plugin is unique amongst the scripting language GUI toolkits, and could be used help make Tk more attractive to (and Tcl more relevant to) other language communities. Finish the installer for Mozilla + Explorer, and get it working on the Mac.
Sharing the lurv
Amongst the developers that we are trying to reach are those who are looking for a GUI toolkit and who would be tempted by Python, Ruby, or Perl or (to a lesser degree) VB if they didn't know any better. And to people already using those languages who need a GUI toolkit.
The positioning in the shorter term is to modernize and relaunch the "Tk story". That is, scripting lets you very cleanly and concisely put together a GUI that's as good as native, with way less effort (we aren't there now, because it can't be native without a lot of picky work, so you have "native" or "concise", not both). When used appropriately Tk is arguably better than the alternatives (Cocoa, wX, GTK/QT, Java/SWT, VB), and will be the “gentle” introduction to Tcl for people in other language communities.
The goal of modernizing and relaunching the Tk story is to reclaim the GUI high ground.
How can this be done? Jeff’s suggestion of updating the Tk bindings to other languages is worth expanding on. This, combined with the better “out of box experience” would be the start of clawing back Tk’s reputation amongst other language communities (Jeff - expand please).
The Tcl browser plugin probably needs to be part of the story too, providing finishing doesn’t divert resources from integrating with other languages. The plugin is unique amongst the scripting language GUI toolkits, and could be used help make Tk more attractive to (and Tcl more relevant to) other language communities. Finish the installer for Mozilla + Explorer, and get it working on the Mac.
History shows that the best software has been developed by small teams who have the skills and are allowed to innovate within boundaries. Accordingly, splitting the Tcl into a language core and several modules would allow for faster evolution as each could have its own team (with a greater sense of “ownership”) and independent release cycle. If you’re going to eat an elephant, don’t attempt it in one sitting.
Strategy
The modules would be those parts of Tcl that could (and should) evolve at a different rate than the core language.
The key is that each module would have it’s own semi-independent development team and release cycle, but retain TIP governance w.r.t. the API version that is considered “stable”. This would allow the module to evolve more quickly (or be intelligently designed ;-)) until the API settles down and people begin to rely on it.
For example, the networking module is crying out for IPV6 and UDP socket support - it shouldn’t have to wait for another full Tcl release rather than progress at its natural rate as features are added. Likewise DavyGravy’s iocp channel support on Windows.
Under this model, Tk itself would become just another module (or collection of modules).
The “Tcl language platform” would be the core Tcl features, plus a snapshot of each module API that the TCT is comfortable incorporating (more on this later). This introduces a distinction between the development and stable versions of a module, but in a way that hopefully would meet the goal of stimulating progress whilst preserving the core values.
Non-TIP managed modules might be included - for example parts of tcllib, SQLite or critcl. This would just mean that the Tcl language platform contains a known, stable version of each - not that they would be brought under direct TCT governance. But think of the benefit for newbies - you get Tcl + a GUI toolkit + an SQL database plus the ability to embed C code if you need to.
Likewise, new modules might be added
The “Tcl language platform” would provide single point to get the “best of breed” or recommended functionality at the source code level. For example,
Note that the initial emphasis here is governance not source management. If the existing source management tools (specifically CVS on SourceForge) aren’t sufficient to support this model then consideration should be given to using alternative tools. Even a best-of-breed proprietary one like BitKeeper (assuming that Larry and the community can work out a licensing arrangement that doesn’t preclude people like Kevin from using it). A future goal should be to separate each module into an individual source trees.
As Joe has put it - the Tcl Language Platform is a set of “modularized components with individual, independent release streams, collected and bundled up in a neat package by the Tcl Language Platform team”.
In summary, as Colin has said ...
“we know Tcl has heaps of value. Others don't. Adding puff, by putting space between the bits, makes it look less like a monolith, and gives outside eyes more to see ...It occurs to me that splitting out the bits, and giving them names, makes the language appear larger. So, Python has 'twisted framework' and that's like rilly rilly cool. Whereas Tcl has that and more, but it's hidden inside the language.
I can envisage/imagine the press release 'Tcl Community Team announces the Tcl Language Platform' make it sound like an expansion of the language, but all it really is is picking apart the threads and giving them individual names. It's really just leavening the mix.”
Any new binary distribution of the “Tcl Language Platform” is going to compete for mind-share with ActiveTcl, whose name is already synonymous with a single download, quality-assured distribution.
Just as it is less than ideal for a newbie to have to make choices regarding widget sets and other extensions, it would be equally unfortunate if there existed multiple binary distributions each purporting to be the best available. It would dilute the opportunity for positive publicity, and fragment the community efforts.
But this needn’t be the case (bear with me here Jeff ... there is some logic to this) - if ActiveState were willing to allow the use of the name ActiveTcl for the “Tcl Language Platform”, and rename the existing ActiveTcl to “ActiveTcl Maxx” or “ActiveTcl Pro” or similar.
That is, ActiveState (or indeed another organisation) could continue to provide the community with a quality-assured binary distribution, that was equivalent to the source distribution used by (for example) Debian maintainers, and was smaller in side than the current largish download. And ActiveState could (if they chose) continue to provide the current more feature-filled distribution for those who require it (presumably, more experienced developers).
Apart from the size, one other issue with “batteries installed” distribution is the slow startup time due to the number of packages (someone clarify please). One way to address this would for the “Tcl Language Platform” to have a mechanism that effectively preloads the “package provide” for each module so that no searching is required on startup or when looking for a package. Further, it might be worth considering Larry’s suggestion of having all standard packages available so the newbie developer doesn’t need to “package require” several packages to get started (recognising this in itself will require a change in the package mechanism so that loading is deferred until needed).
This does imply that the ttk and tk widget sets need to be reconciled in some way, ideally in 8.* classic.
Whilst acknowledging that the two widget sets are complementary, the current situation exacerbates the “I don’t want to choose, just give me the right thing” negativity confronting newbies (and more experienced programmers for that matter) who just want a good look or feel out of the box without having to package require ttk or tk or whatever. Classic Tk and/or Ttk could still be modules, but the developer should not be forced to choose them.
It would be ideal to aim for release of the new Tcl/Tk book concurrently with 8.6 - this - this would help promote both the book and the release, although it does mean 8.6 feature freeze is needed very soon (this shouldn’t be a problem as I understand it from Donal). Ken - is this feasible?
Tcl and Tk are perceived as dated and dying by many outside of the Tcl community. Whilst we know Tcl/Tk has many compelling features (event driven I/O, a safe threading model, internationalization, portability, flexibility, easy to build GUIs, etc) it is nevertheless a “hard sell” due to baggage the Tcl name carries from the past, plus the dated look and feel out of the box.
The result is Tcl/Tk is perceived as “old hat” and there hasn’t been an injection of new blood, with new ideas and more people to share the load.
How can we address this? By a new Tcl and a new Tk - both made possible by the stability of the Tcl + Tk classic releases.
The new Tcl should
The new Tk should leapfrog the competing cross-platform toolkits by providing a high-level Tk-like language to access functionality available on modern systems, including video and audio. Ideally this would involve a new (and possibly complementary) toolkit based on the NexTk research work being conducted by George, SteveR, Arnulf and others.
Given the NexTk work is already underway and is a ground-up rewrite, and there is no direct effort from Jeff, then it may as well be formalised straight away. The benefit is mostly symbolic - in that it endorses and encourages what these people are doing already, plus sets the expectation that it can become mainstream one day.
The new Tcl work on the other hand, could wait till Tcl 8.6 is complete - so as not to dilute focus of Tcl maintainers from getting an 8.* classic release.
As mentioned (forcefully) by Larry, backward compatibility is a reasonable goal but is preventing progress in a number of areas.
The New Tcl language core (a.k.a Tcl 9) is the time to break this compatibility where appropriate - not gratuitously but to remove the accretion of nearly two decades years of development and allow those with new ideas (e.g. Joe, Miguel and others) to take the language forward.
There are two possible paths to a cruft-free, legacy-free codebase - one way is to clone Tcl 8.5 and start tossing out the cruft. The other is to start afresh, and copy/paste/cleanup bits from the old codebase as needed. Joe offered an interesting suggestion - do both in parallel and whoever gets there first wins :) But perhaps a combination of these two approaches will work well - decide at a high level which features are retained, and then use whichever approach is more appropriate when implementing. The key is to free up
the Tcl maintainers to improve the codebase (and have fun doing so) without having to worry about backward compatibility - this is a “once in a generation” opportunity.
It is anticipated that most of the incompatible changes would occur at the C API level, and that scripts would be mostly compatible. But Joe has raised the point that features should be considered for inclusion, rather than taking all existing features and say “which ones should we omit” (Joe, please expand).
Jeff / Joe / Miguel / Donal / whomever please expand on incompatible changes you are hanging out to make
To get the process started, Tcl 9 could be nothing more than Tcl 8.6 + TIP 114 (DieOctal).
That doesn’t mean there has to be a formal 9.0 release, but there does have to be an announcement and a source tree that can be downloaded and tried. This should happen sooner rather than later, especially given that Miguel, Joe, George and others are already experimenting with VM performance.
In addition to the source distribution, it would be advantageous to have a regular starkit build for common platforms - to make it easier for people to try out new features without having to “install” a distribution that might collide with their existing Tcl/Tk installation.
There should also be a continued application of the previous development model where multiple project teams are encouraged to work on areas that interest them.
Colin has suggested a couple of other features that would be worth considering - TAL (Tcl Assembly Language) and a Tcl DOM for Firefox.
The Tcl Assembly Language - direct bytecode generation from tcl - would allow command compilation to be done in tcl, not just in C. The value proposition - it's a kind of introspection, it's unique, it's eating our own dogfood, it's low hanging fruit (not hard, technically) and it's a buzz technology.
The idea of a Tcl DOM for Firefox could end up being a disruptive technology, and is made feasible by recent developments (Colin - please expand). And there is also the recent Tcl+javascript work.
Other ideas that people are keen to work on?
The New Tcl needs to provide a direct benefit to users, not just core maintainers. But rather than just play catch-up, one or two “hot” new features might just enable some “buzz” to be generated.
What should this be?
Jumping to conclusions - it seems like a hot new feature in hardware is multi-core CPUs. Dual core is the norm, 4-way is common, and one can assume within the next few years even 64-way is going to be common.
Almost uniquely amongst scripting languages, Tcl is well positioned to take advantage of this due to it’s compartment threading model. As Joe mentioned, there might be something we can learn from Erlang (e.g. a fast thread clone operator) but mostly it is there. On top of that abstract a message passing model along the lines Donal discussed (not just multi-core but multi-host as well) and you’ve got a pretty good story.
The following notes are courtesy of Donal ...
The first (and probably most important) is unicast, where there is one sender, one receiver, and the sender knows the receiver at the start.
The second is broadcast, where everyone receives the message and the sender doesn't need to know anything about the recipients. This is not that useful in practice (except for stuff like DHCP) since it scales terribly, and hardly any messages are interesting to everyone.
But there are other cases that are between these two that are much more interesting. Here's a zoologists guide:
Multicast: like unicast, but the message goes to many recipients. Analogous to having an email message with multiple people in the To: field.
Message Board: there's a "topic" that the sender can post to, and the first interested receiver takes the message off. Difficult to scale. Compare with tuplespaces. Requires a rather sophisticated message broker layer to work IIRC.
Publish-Subscribe: there's a "topic" that the sender can post to, and all interested receivers get copies of the message. This sort of system is used by IBM's MQseries, and there's an analogous product from Microsoft. Compare with tuplespaces. Also note that TCP multicasting is really a variant of pub-sub, where the topic is the multicast address.
There appears to be a lot of smart money riding on pub-sub.
In terms of reliability of message dispatch, the way to do it is to first deliver the message into an outgoing queue database that is local to the sender. Then use a distributed transaction (or the TCP "shout until they tell you to stop" approach) to copy the message to the receiver's incoming queue DB. Only once it's made that hop right does the receiver actually get the message delivered to it. (And even with all this, it is apparently very hard to get Once And Only Once Delivery semantics.)
Native support for pub-sub messaging would make it very easy to construct sophisticated programs, and they should be fairly easy to distribute over a cluster.
Speaking of which, if we could have a version of Tcl+Thread which could work with multiple processes (especially if communicating using something like MPI* for the IPC fabric) that would get a lot of traction in the modern scientific world. Currently they mostly use Python (and Fortran, of course).
* Tcl does not do OpenMP at all, as it requires a shared memory model. Ignore it.
Add to that a discovery implementation based on a Zeroconf implementation such as Bonjour [link] and you have a way to do message based parallel computing across multiple-cores and multiple hosts using a publish-subscribe model. Of course, Bonjour is more suited to local networks, but it has been sufficient for products like iTunes, Skype, Gizmo, Asterisk, Adobe Creative Suite to use it for local service discovery - so adding Tcl to those wouldn’t hurt.
SteveR / George - please insert some stuff on the NextTK project - specifically a high level overview of the rationale, design goals and approach as per our discussions.
It will be important to get Daniel, Joe and Pat involved so that cross-platform issues are addressed (my understanding is that the work to date has been done on X11 primarily).
Increasingly, there are quite good cross-platform C libraries that abstract system functionality, including:
In addition, common Operating System platforms such as MacOS X and Windows provide compatible system functionality (e.g. audio, video, networking).
One of the real strengths of Tcl has been as an integration language. The reality today is that is still true if the technology you're trying to integrate with was built in 1995 or earlier. In some respects, Tcl has gone from a "glue" language to a "dried-up glue" language.
But this is changing - Critcl has been used to wrap this functionality behind a higher-level cross-platform API and make it available as a Tcl package. For example, the muzic package wraps fluidsynth, CoreAudio and QuickTime on OSX, and winmm/dsound on Windows.
As part of this exercise in making Tcl more relevant would include a team that extend Critcl’s wrapping abilities (at least, by allowing structs to be returned as dicts and vice versa) and look for opportunities to wrap such cross-platform libraries and system functionality. This would, for example, facilitate the production of a system module.
This might also include a small C compiler such as tcc (reference Mark Janssen’s recent work) - to act as a fallback if there wasn’t a toolchain installed on the current machine, and also to facilitate cross-compiling to Windows on *nix platforms.
The TCT/TIP process has been focussed on a tightly managed Tcl “core”, with a single major release cycle. As has been discussed, this delivers stability but not rapid progress.
If the “core” is divided into multiple projects, the role of the TCT would need to change - it would focus on the language “core”, and there needs to be a new mechanism to determine what goes into the “Tcl language platform”.
As Larry mentioned, a barrier to newbies is that there is too much choice. The language platform needs to contain the a “best of breed” of each component - be it widgets or toolkits or library code. This doesn’t preclude other modules being available, but a newbie coming to Tcl isn’t forced to make decisions before getting started.
One way of doing this would be a new community panel (I hesitate to use the word committee) - endorsed by the TCT, and comprising a significant “user voice” (e.g. Brian G or Steve R).
The role of this panel would be to
Perhaps more importantly, the panel would decide what doesn’t go into the language platform - thereby reducing the choices that a newbie must make when coming to Tcl. This will inevitably alienate some people, but such is life.
Larry has raised concerns about committees and group decision making. To reduce the risk of slow progress the panel needs to be kept relatively small and focussed, and have a "head" who can make things happen - select panel members, coordinate the panel, set goals, act as tie-breaker if there are disagreements, liaise with the TCT. And that head should be "blessed" by the TCT - to achieve continuity and credibility with the existing Tcl/Tk governance model.
Need something addressing
An aging and declining population needs to both get more out of the people who are already there, but also foster immigration. :-) One group of people worth bribing are the leaders and major commiters (if any exist) for the Ruby, Python and Perl Tk bindings. Having them on board would play huge dividends. Bringing them to the conference, or a special meeting specifically about Tk, and picking up the entire tab would be a very good use of cash (thanks to Mark Roseman for this idea).
It would also be good to have a register of skills available, and people with needs. And a means for people to sponsor specific projects - not everyone in the community can work for dollars, and a non-financial means of providing incentive might work well (for example, SteveR supplied one community member with a number of books as an incentive).
Seriously, it isn’t. At least for people whose businesses depend on Tcl.
We need to market Tcl so that it is no longer the industry’s best kept secret. There are too many places to find info about Tcl - there needs to be a consolidated effort behind a single, contemporary face - addressing:
Companies basing their products on Tcl/TK
George McK recently floated a concept worth considering: That ten commercial developers of applications using the Tcl/Tk technology purchase the mini.net domain and utilize it to promote and cross- promote their products in a simple consolidated manner to non technical people.
The site to simply consist of a home page with; explanatory text of Tcl/Tk technology ; a revolving listing / linking of the ten commercial developers and their applications ; and a press release section. (Further items in that vein may be added as marketing progresses).
The audience would be non technical yet professional people in other spheres of activity (Finance, HR, Sales General Management). The benefits would be the cross-exposure for the commercial developers, leading to sales and usage, for a small amount of money.
There are obvious benefits of scale to be explored and enjoyed at a small cost.
Regular articles highlighting unique or interesting features
To get the word out, but in a contemporary communications media - e.g. Michael Cleverly's blog at [link]
Posted at Dec 31/2007 06:33PM:
comment from the chat from tclguy - so perhaps the next "marketing" solution is to assist with key projects (coccinella, amsn, git-tk ...???) to make them top-notch 8.5-based tools?