CodeMonkey

context: January 2005 is 4 months earlier than the latest web.archive of Clicker website. It is still lacking important (?) pages such as CodeMonkeyInternals (restructured from the current page ?) or ResourceCmPlugin explaining another thing code monkey was used for.

---

The CodeMonkey is a [Tool] for code rewriting that is due for Clicker 0.8.20 It will helps writing KDS/IDL code, accessing koLib templates (foreach on koList, cursors, etc), defining koLib & memory classes and much more.

Mechanics of CodeMonkey

The CodeMonkey core is a Perl script that performs input tokenization and manage context stacks (each 'block' has its own context). It comes with a helper package Tokens::Filter.pm that allows one to easily perform high-level pattern matching against operators, etc. Each CodeMonkeyPlugin can be requested separately via the @plugin "name.pl" in the source file.

Rules to write CodeMonkeyPlugin~s

All the 'top-level' code is for initialization

Among other things, it should insert _keyword hooks_ in $context->{hooks} to have plugin functions called when a keyword is encountered. You can also register _terminators_, that is, plugin functions to be called when the input file is completed.

# a skeleton CodeMonkey plugin

use Tokens::Filter;
print STDERR "using skeleton plugin\n";
$context->{hooks}->{skeleton}=['SKELETON',\&myHookFunction];
push @{$context->{terminators}},['SKELETON',\&myTerminator];
true;

sub myHookFunction {
}
sub myTerminator {
}

Hooks receive $mode, $tagname, $codeA, $codeB

The $mode parameter tells you the context where the keyword has been seen. It could be an 'INSTR' or a 'BLOCK'. $tagname returns you the first word that was associated with the hook (here 'SKELETON'). This can be used when multiple similar keywords wish to use the same function but still differenciate the cases.

$codeA and $codeB contains what you need to pass Tokens::Filter to analyze the code that triggered the hook.

sub myHookFunction {
  my ($mode, $tagname)=@_;
  my $code=new Tokens::Filter(@_[2,3]);

  # write your stuff here
}

$context and blocks

Each block construct has its own $context information. Thus when your hook is registered with $mode eq 'BLOCK', you can register local hooks in $context->{hooks}, etc. The "header" of the block (e.g. all the text between the previous instruction/block and the { symbol will used for the "block name" (after tags are processed). A hook can discard the name's output by setting a true value in $context->{naked}

$context->{upper} chains towards the upper-level context information.

controlling INSTRuctions

The hook can decide whether or not a terminating ';' should be written after the instruction's translation by returning a _true_ or _false_ value.

matching text

The Tokens::Filter package can be used to check high-level patterns in the 'running code'. The function to use is $code->match(<offset>, <pattern>). The <pattern> is a list of item types describing what we expect to be on the input:

• WORD matches an identifier that is not a registered keyword.
• LITT matches a litteral (e.g. a string/character)
• TAGG matches any identified keyword
• OPER matches any sequence of non-alphanum and non block-forming characters. Note that both "+","++","=++","*=&" will be seen as valid OPERators.
• SLST matches an opening parenthese
• LIST matches a completed list (pointing towards a specific code sequence).
• THIS matches the current tagged-keyword
• TRAN matches a translated item (from a previous hook application)

The type can be followed by a ":<value>" string further restricting possible matches For instance qw(THIS WORD OPER:~= LIST) will match an code sequence like skeleton var=(any thing can be here). The <offset> argument of $code->match tells the relative position of THIS in the pattern.

# let's try to check if we have the expected "skeleton var = (...)" framework.
sub myHookFunction {
  my ($mode, $tagname)=@_;
  my $code=new Tokens::Filter(@_[2,3]);

  # 0 is the position of "THIS" in the list ...
  if ($code->match(0, qw(THIS WORD OPER:~= LIST))) {
     # here we know.
  } else {
     die "unexpected use of 'skeleton'".$code->show;
  }
}

getting/replacing text

Once a pattern has been matched, you can get the text of the different items using $code->content. Each item in the pattern is numbered from _0_ to _N_ and you can retrieve its value with $code->content(i). For instance,

  # retrieve the variable name and the content of the list.
  # note that we get the list content as a raw string.
  # "skeleton myVar = (a,b,c,d,e)" ==> $variable_name eq 'myVar' &&
  #   $list_content eq '(a,b,c,d,e)'
  my $variable_name=$code->content(1);
  my $list_content=$code->content(3);

As the purpose of CodeMonkey is to _rewrite_ text, we could like to replace the actual text by something else. That is (again) done with $code->content() giving the new value as an additionnal argument:

  # rewrite it as "int* myVar[]=..."
  $code->content(0,'int*');
  $code->content(3,'[]=');

Of course, you may use PERL's power for more complex operations, for instance if i wish to have all the items in the $list_content translated so that we extract their address and make an array out of it, it simply means

#  remove parenthesis
$list_content=~ s/^\((.*)\)$/$1/;
#  split the content, assuming a flat list
@list_content= split /,/,$list_content;
@list_content= map { '&'.$_ } @list_content;
$code->content(2,"{".join(',',@list_content)."}");

#now we have "int* myVar[]={&a,&b,&c,&d,&e};

---

Date: Fri, 14 Jan 2005 06:52:43 -0800 Mime-Version: 1.0 (Produced by PhpWiki 1.3.9) Content-Type: application/x-phpwiki; pagename=CodeMonkey; flags=""; author=PypeClicker; version=3; lastmodified=1105714363; author_id=PypeClicker; markup=2; summary=more%20like%20a%20tutorial; hits=124; charset=iso-8859-1 Content-Transfer-Encoding: binary

ClientCmPlugin

Context: I thought I had lost the clicker wiki forever, or at least, that I couldn't recover the part describing the late "code monkey" mechanics I had developed and used to "ease" development. It seems like some part of it had been saved in some wikidump folder in an obscure location of rsync-only file server of sourceforge. Maybe it isn't the latest version, but let's have a sample of what's in there anyway...

---

client.pl is a CodeMonkeyPlugin that performs rewrite of [KDS] client/server code, much like SlangSyntax used to do, but taking greater benefit of [IDL]-generated knowledge

Declaring interface we'll use

Each interface will use a *prefix* for its identification. It's important that one give different prefixes for different interfaces, especially if there are methods named the same way in the interfaces. Interface declaration can use either client or using keyword depending on whether you also want a struct kdsClient to be created.

syntax:

using "$servicename$":$interfacename$ as $prefix$;

client "$servicename$":$interfacename$ as $prefix$;

$servicename	the path from kds://services to the kdsService (e.g. timing, sys.paging, sys.binterpreter, dev.disk, etc)
$interfacename	the name of the declared interface on the service, just as in IDL files
$prefix	must be a C-compatible token that will be used to prefix every interface-related things like messages structure, etc. The prefix may vary from one source file to the other.

using and client statement superseeds the need for #include <api/___.api> and should precede any use of the interface (either through implementation or invokation). Moreover they must appear at top-level.

declaring a simple server

The server declaration will need either using or client to be first defined. Each server may implement any number of interfaces but they should all belong to the same service. Server declaration must also appear at top-level

syntax:

server "$servicename$" { (<server_command>|<implementation>)* }

<implementation> ::= implements $interfacename$ { <method>* }
<method> ::= method $methodname$ ( $serverinfotype$* $serverinfovar$, message $name$ ) { <code> }

the method command will generate the appropriate function prototype, using _$prefix$_$methodname$ as function name. You normally don't need to know that name unless you want to do funny KDS bypassing stuff. Functions parameter are available through the message structure (e.g. $name$->$parameter$)

Adding ServerCommands

Since 0.8.20, the "client" plugin is also able to handle server initializaion/termination and activity callback declarations. The syntax is

<server_command>::= <server_vars>|<server_methods>
<server_methods>::= on $eventname$ ($args$) { $code$ }
<server_vars>   ::= with (queue|thread) $varname$ = <value>;

A sample

Let's suppose we defined a 'test:hello' interface with methods void hello(char* who); and void bye(char* who);

@plugin "client.pl"
using "test":hello as greet;
client "display":basic as print;

server "test" {
   implements hello {
      method hello(void* we_dont_care, message m) {
         printStr(&DefaultConsole,"Hello %s!\n",m->who);
         return KDSE_OK;
      }
      method bye(void* we_donT_care, message m) {
         printStr(&DefaultConsole,"L8r, %s...\n",m->who);
         return KDSE_OK;
      }
    }
}

---

Date: Tue, 19 Oct 2004 08:34:12 -0700 Mime-Version: 1.0 (Produced by PhpWiki 1.3.9) Content-Type: application/x-phpwiki; pagename=ClientCmPlugin; flags=""; author=PypeClicker; version=3; lastmodified=1098200052; author_id=PypeClicker; markup=2; hits=49; charset=iso-8859-1 Content-Transfer-Encoding: binary

Coding Kick-off Meeting

Memories of a meeting that could have been helpful when we were a team working on Clicker. It actually happened with a EU project a bit later

• it was our ordinary 3-days-midweek meeting
• almost no professors involved, but everyone who would program anything for the project was present
• we came with machines ready to build the equivalent of a 'hello world' for the feature we're targetting. For Clicker, if we were to build a sound mixer, 'hello world' would have been being able to output a square wave. If it was to make a configuration tool on a foreign OS, 'hello world' would indeed be a window and an 'Hi.' button that tells it to proceed
• we would not try to have a polished outcome (that would be the job of a smaller, dedicated team, starting after the meeting) but to identify all as many as possible of the issues that could block the smaller team and prototype a solution to them. For the sound mixer, applying volume per channel could be left post-meeting, but deciding how to tell the channel's volume from a control pannel has to be prototyped. For the configuration tool, reporting ongoing progress bar must be prototyped, 
• people don't need to know the same technologies. If one coding team will use ncurses while another uses WxWidgets and two others go for electron, that's fine.
  

Now imagine if I had heard of that 'sprints' back then:

• Every morning, we spend one hour deciding what we are trying to achieve by the end of the day, what are the goals that are candidate to prototyping and which should get our attention first
• Each coding team spends 4-6h trying to prototype what has been decided
  
• (my 2 cents: If one team is done faster, they can use the remaining time to study the technologies picked by others, how they address what they've just done with that, rather than trying to tackle more objectives)  
• The end of the day is used to review what has been achieved, identify strengths and weaknesses of each approach, and update the list of candidate goals for the next day. "Oh, that makes me think: we'll definitely need a way to pick a file for ${purpose}". We'll see tomorrow morning if that is important enough to be one of the prototype goal of the meeting.

(Hopefully, at the end of the meeting, everyone has a better understanding of each others' skill with their technologies and some of the weaknesses of the technologies. Hopefully enough to decide the team and technology to actually implement the features)

gdbm perl tools ... could they save the lost wiki ?

Clicker was the first of my projects to use a Wiki. And the last time I thought about an "information browser" program. Yet, the phpwiki database was broken on a regular basis. I built gdbmpatch and gdbmshow helpers to know how I could repair it. The Clicker project sort of died when a last update to the sourceforge policy made the phpwiki no longer working.

#!/usr/bin/perl

# a wiki file is made of different keys for each page. The content is 
# a sort of "bencoded" file with following rules:
#   * s::"" encodes a string
#   * a:<#items>:{<;-separated content>}
#   * i:

# p contains the html cache of the page (compressed)
#   available keys are $_cached_html and !hits

# li contains "backlinks" as a simple array (keys are integers, values are page names)

# lo contains "page links", same structure backlinks

# v: contains one of the page's version.
#   $author and $author_id tells who wrote the page
#   $summary, !mtime tells more about the page.
#   $pagetype should be "wikitext" and "%content" is the whole content.
# note that if version i exists, versions 1..i-1 should exist too.
# obsolete versions have an additionnal "_supplanted" key.



use GDBM_File;
use PHP::Serialization qw(serialize unserialize);

tie %file, 'GDBM_File', $ARGV[0], &GDBM_WRCREAT, 0640;

print "tied $ARGV[0]. items: ".keys(%file)."\n";

delete $file{pResources};

untie %file;

#!/usr/bin/perl

# a wiki file is made of different keys for each page. The content is 
# a sort of "bencoded" file with following rules:
#   * s::"" encodes a string
#   * a:<#items>:{<;-separated content>}
#   * i:

# p contains the html cache of the page (compressed)
#   available keys are $_cached_html and !hits

# li contains "backlinks" as a simple array (keys are integers, values are page names)

# lo contains "page links", same structure backlinks

# v: contains one of the page's version.
#   $author and $author_id tells who wrote the page
#   $summary, !mtime tells more about the page.
#   $pagetype should be "wikitext" and "%content" is the whole content.
# note that if version i exists, versions 1..i-1 should exist too.
# obsolete versions have an additionnal "_supplanted" key.



use GDBM_File;
use PHP::Serialization qw(serialize unserialize);

tie %file, 'GDBM_File', $ARGV[0], &GDBM_WRCREAT, 0640;

print "tied $ARGV[0]. items: ".keys(%file)."\n";

foreach(keys %file) {
  next if !/$ARGV[1]/;
  print "$_ ==> $file{$_}\n\n- - 8< - -\n";
#  delete $file{$_};
}

untie %file;

Drop the question mark ...

In its original design, and up to version 3.0 distributed with the 3rd edition of his book (2005), Andrew Tanenbaum's MINIX was a single-address-space operating system. Granted, you could have multiple process running simultaneously and they're isolated from each other, but they all happily frolic in the same address space, and only the segmentation unit of the x86 processor prevents the havoc from happening.

It's not necessarily a bad thing: paging introduces overhead in address resolutions - especially when your virtual-to-physical translation buffer is no longer sufficient, requiring up to 3 cache misses before you get a single byte of data. Its impact on context switching is even more frightening than this: whenever you wake up another process, the whole translation buffer has to be flushed (okay, *some* pages -- usually those holding the kernel -- can remain sticky).

Not having paging has a huge impact, however: you can't build any sort of modern virtual memory. No partial swapping of some unused part of the software... and no "statistical allocation" of the memory. Much like in MS-DOS times, if your compiler *could* need up to 16Mo of RAM, you must give it all from the start. If it happens to need only 8Mo for the file you're compiling and that you need those other 8Mo for doing something else meanwhile, well, too bad for you.

But it has changed since then. Between 2008 and 2010, people have been adding a "virtual memory server" to Minix, which in turns remembers me of my "pager2" service in Clicker. It let you control mapping of your address space (do_mmap, do_mmunmap) and supports the process manager (do_fork, do_exit), but here, it also directly handle page faults through message passing. Again, it's very "memory object"-like (another key Clicker concept), which shouldn't be a surprise: I got the idea of the memory object while reading the other Tanenbaum book about operating systems :)

Altogether, I'm absolutely not convinced that placing the address space management into a server rather than into the microkernel was the best design choice one could make. Granted, 'forking a process' is something that could happen out of the microkernel, but pagefaults ? ...

Iirc, I had the idea of letting the Clicker microkernel know that some physical pages had been 'pre-allocated' to some memory object (which you can think of as 'virtual regions'), and only notify the server-level code about a miss when that pool is exhausted... a sort of hybrid micro/exo kernel. But I dropped the whole project before I got to that point.

One of my major "errors" regarding the Clicker overall design, though, was that will to make the paging optional, despite it turned out that every other feature was depending on it. My module mechanism allowed me to do so, but it over-complicated the whole code, introducing the need for sophisticated book-keeping structures, run-time service replacement, and a mysterious "private heap" feature where linker scripts should have been enough.

the lost Wiclicker

Back in June 2004, I stopped trying to have a documentation as static .html files, and turned towards a (php)wiki for Clicker documentation and research. It turned very useful to coordinate our efforts as DasCandy provided some sporadic help here and there, and whyme_t developped installer tools and modules for Clicker. I was so happy to see a team at least starting!!

The wiki is now unfortunately defunct after both a spam assault and my failure to identify a sudden bug in Php code. I have a last dump on my system, but I haven't managed to recover the content yet.

After the lost forums, it was somehow too much to try to recover things at all.

Hyper-Desktop Markup Language ?

Among the "funny ideas" I had for Clicker, many has echoed in other's mind and some eventually got implemented here or there.

I won't call myself as "spolied inventor of tags", of course. I'm not. Though I have to admit tags are 100% aligned with what'd have pushed for Clicker if I had any resources to make things move by just pushing them.

Now, let's have a look at the box #7 of this "clicker desktop mockup" I made up somewhere near 2K++ ... the "one-key-command-line" for quickly spawning things is now mainstream ... Even Windows has it (or at least as some plugin) where you type [ESC]word[ENTER] to search and launch your report editor rather than crawling through clobbered menus.

"Incoming" (last imported documents) and "favourite" meta-folders are of lesser significance given we've got the "download history" window of Firefox. But let's check out that box #7 ... It was claiming "users do tasks, more than they use applications". That thing is starting to change as well, the welcome panels of Thunderbird 3, Wireshark (Lucid release) and K3B (a while ago) being an obvious example of it. They also claimed "a task is performed using a collection of tools operating on a collection of documents". We haven't got anywhere near that so far, afaik. Do I want to integrate something to the wonderful Tomboy application ? I have to learn C# ... Do I want to have Gimp able to learn new key combos for filters ? I bet I'll have to dig into some GTK+ and maybe some scheme or Python would help me.

Yet, all I'll be doing somehow is moving boxes around, connecting wires between blocks of code, etc. This is something that should be as easy to do as writing HTML, but there doesn't seem to be any Hyper-Desktop Markup Language around.

-- edit -- Btw, I was about to attach a picture in a thundermail while thunderbird crashed at me because I was also moving directories around. *sigh*. Long is the road.