Ceci n'est pas un titre

Open Letter to Online Game Developers: Allow Bots

2012-04-06T14:14:00.000+02:00

As someone which a strong interest (and little experience) in artificial intelligence I have a short and to the point request for online game developers: allow bots.

Either have specific servers or, if you're single sharded, places in your game universe where bots are allowed. Outside of those servers and places continue your battle against illegal macroing, gold farming and botting:

In the bot-legal places do allow human players.
When making a new account (or avatar) have a player flag it as being used by a bot. It should still be able to be used by a human as well.

There's huge potential if people are able to legally run their AI projects in a gaming universe. Witness the popularity of AI programming competitions like the Google AI Challenge, the Mario AI Competition, Starcraft AI Competitions and many others.

It would so be nice to be able to experiment with AI bots in online games without the fear of one's account being banned (and having to spend money on a new account).

This blog post is partly aimed at Blizzard since they represent such a large part of the PC online games market with Starcraft 2, World of Warcraft and, in a few months, Diablo 3.

Ghetto DNS

2012-01-06T12:07:00.000+01:00

This is probably stupid, but…

I start my EC2 instance only when I need it but a while ago it was needed pretty often and figuring out the IP every time I wanted to SSH to the machine got annoying quickly. Using the Dropbox Uploader bash script it was easy to set up a poor man's DNS since Amazon provides a way to figure out the public IP of an EC2 instance.

So by doing:

curl -s http://169.254.169.254/latest/meta-data/public-hostname

and using Dropbox Uploader one can put that result into a text file (say ec2-hostname.txt) in a Dropbox folder and use it in an SSH command as follows:

ssh -i your-amazon.pem username@`cat ~Dropbox/ec2-hostname.txt`

This solution had a certain gum and wire charm that I had to try it out but it only suffices for one man in his attic. If you want robust and scalable solution you'll need to use real DNS or Amazon's Elastic IP.

VirtualBox Common Lisp Environment for Google AI Ants Challenge

2011-10-19T11:23:00.005+02:00

(see the end of this post for remarks I've gotten about the vbox image)

So, the next Google AI Challenge seems to be almost ready to get going, albeit six months later than I expected.

As blogged about earlier this year I have been worked on the Common Lisp starter bot for the challenge, an alternative starter bot with a proxybot, some YouTube videos and now a VirtualBox image with an Emacs+Slime environment ready to go.

Due to the long delay for this challenge to get started some of these projects aren't quite up to date anymore but they should still be quite usable and at least get you started or help you along.

The VirtualBox image is up to date as of writing this post and any comments on it would be welcomed. I will not make a new image (although someone else is free to do so) but if possible I will provide a patch on the AI Challenge forums.

On booting the image it will start up SBCL, Emacs and Slime (connected to SBCL) with some instructions in the scratch buffer. Perhaps it is a good start for another Lisp-in-a-box project.

I am not sure how much time I will be able to spend on the current challenge due to also participating in the online Introduction to Artificial Intelligence and starting a three month sabbatical gig writing an iPad client for a startup November 1st.

Specific Remarks about the VirtualBox Image

14:23 < Xach> the remapping of [] and () confused me though.

Right, I've been computing like this for so many years that I totally forgot about this. The parens have been swapped with the brackets otherwise Lisp programming becomes quite painful. If you want to reset this check out keycodes 18, 19, 34 and 35 in ~/.xmodmaprc and reboot the image.

Now that I think of it, Control and Caps Lock are probably swapped as well. That's done in the ~/.xmodmaprc as well.

Adding Text to PNGs

2011-09-25T22:23:00.001+02:00

(I'm putting this out here since Google queries for this situation didn't turn up anything useful for me.)

Last night I had to add some text ± 5000 PNG files and I knew about Zach Beane's ZPNG package for reading PNGs but I didn't know of a package that would read them.

A quicklisp:system-apropos turned up png-read written by Ramarren which looked simple enough to use so I went with that. After initially failing to get ZPNG to use png-read's image-data slot I gave up and wrote a little loop to copy the values over to zpng:data-array. This was succesful after one or two tries:

(asdf :png-read)
(asdf :zpng)

;; http://imgur.com/qtriH.png
(defparameter png (png-read:read-png-file "qtriH.png"))
(defparameter zpng (make-instance 'zpng:png :color-type :truecolor
                                  :width (png-read:width png)
                                  :height (png-read:height png)))

(loop for y from 0 below (zpng:height zpng)
      do (loop for x from 0 below (zpng:width zpng)
               do (loop for rgb from 0 below 3
                        do (setf (aref (zpng:data-array zpng) y x rgb)
                                 (aref (png-read:image-data png) x y rgb)))))

(zpng:write-png zpng "tmp.png")

I'm sure Zach had a good reason to reverse the x and y in ZPNG but I can't deduce it.

So, reading and writing the files was working but now I realized ZPNG didn't have any functionality for generating text. (Why should it? It shouldn't.) I started thinking of Vecto since I had used that before and also recalled that it actually used ZPNG for saving PNGs. Now all I needed to do was getting the original image into Vecto so text could be written over it.

Going through Vecto's source showed me I could get at the ZPNG object through the (shadowed) *GRAPHICS-STATE* global. It wasn't exported from the Vecto package but since I was in hack mode anyway it didn't really matter. I used the (slightly adapted) code above, drew some text, did a VECTO:SAVE-PNG and got an empty image.

In Vecto's source I noticed the ZPNG object had four channels instead of the three I was using so I set the fourth channel (alpha) to opaque and got my input PNG back with the custom text generated by Vecto. Done!

(asdf :png-read)
(asdf :vecto)

(use-package :vecto)

;; http://imgur.com/qtriH.png
(defparameter png (png-read:read-png-file "qtriH.png"))
(defparameter zpng (make-instance 'zpng:png :color-type :truecolor
                                  :width (png-read:width png)
                                  :height (png-read:height png)))

(with-canvas (:width (zpng:width zpng) :height (zpng:height zpng))
  (loop for y from 0 below (zpng:height zpng)
        do (loop for x from 0 below (zpng:width zpng)
                 do (loop for rgb from 0 below 4
                          do (if (< rgb 3)
                                 (setf (aref (zpng:data-array (vecto::image vecto::*graphics-state*)) y x rgb)
                                       (aref (png-read:image-data png) x y rgb))
                                 (setf (aref (zpng:data-array (vecto::image vecto::*graphics-state*)) y x rgb)
                                       255)))))
  (set-font (get-font "font.ttf") 32)
  (draw-centered-string (floor (/ (zpng:width zpng) 2))
                        (- (zpng:height zpng) 80)
                        "Hello, World!")
  (save-png "tmp.png"))

What has happened to David O'Toole?

2011-07-31T11:22:00.003+02:00

Update: he linked me to this restatement on #lisp: http://cryptome.org/0005/dod-lisp-sol.htm

Does anyone know what has happened to David O'Toole? He hasn't been heard of since a last curious post to the lisp-games-dev mailing list: http://lists.common-lisp.net/pipermail/lisp-game-dev/2011-July/000158.html (read the whole thread)

His website, blog, twitter and github have all been either removed or cleared:

Common Lisp Proxy Bot Videos for Google AI Ants Challenge

2011-07-23T20:54:00.001+02:00

I have made two screencasts showing the setup and usage of the proxy bot for the upcoming Google AI Challenge. This was at the request of McLeopold.

For the Planet Wars challenge I've tried to explain the use case for a proxy bot on the forums. Please read that post if you have no idea what the proxy bot does, although I will try to sum it up now: the proxy bot allows one to develop your actual bot in the usual 'Lispy' way[1] while the game engine runs and kills the proxy bot.

Since it uses sockets for communication it ought to work for other languages than Common Lisp but this hasn't been tested.

[1] Running Lisp image and, for example, Emacs + Slime.

Ogre3D / Okra Update

2011-04-05T15:16:00.000+02:00

Okra (Common Lisp bindings for the Ogre3D engine) hasn't seen a lot of work lately. It covers my current needs and I've mostly been off to other projects. However, there's good news from the Ogre camp: there's now a community effort to create a C API!

I can only hope the API will have settled down before I return to using Ogre.

Preparing for the next Google AI Challenge

2011-03-31T10:57:00.001+02:00

Work is in progress for the next Google AI Challenge and I've been working on the Common Lisp starter package. At the request of the organizers the starter package has been made very basic.

A more extensive starter package with proxy-bot functionality is available at: http://github.com/aerique/google-ai-challenge-2011-1-ants/.

Proxy-bot functionality is meant to allow for more Lisp-like development of your bot. You can keep your actual bot running in, for example, a Slime session while the proxy-bot is started up and killed by the game server. The proxy-bot connects to your actual bot in the Slime session to play the game, ie. it just passes on state and orders between the game server and your actual bot.

Baby Steps into Genetic Programming

2011-01-18T16:39:00.002+01:00

1 Introduction
- 1.1 Toy Project
- 1.2 Environment
2 Generating Random Code
3 Running Generated Code
- 3.1 SBCL note
4 Population
5 Fitness
6 Generation Functions
7 Advancing a Generation
8 Conclusion
9 Thanks

1 Introduction

While my final ranking in the Google AI Contest was disappointing (280th), it was a very educational experience and was totally offset by Gábor Melis' dominating win using Common Lisp as well.

One of things that piqued my interest during the contest was a post on the AI Challenge forums about a bot written using genetic programming. Genetic programming (GP) and genetic algorithms have always held my interest but seeing the bot in action really motivated me to dive into the matter.

Genetic programming is inspired by biological evolution and is a way of solving problems by setting up an environment (tuned to the problem at hand!) and allowing computer programs to evolve towards a possible solution in that environment.

This article shows my initial exploration into GP using Common Lisp and should be an example of a typical REPL session (my session was a couple of hours divided over two evenings). The code has been reviewed, made a more readable and lispier but still looks very much like what I wrote initially. From the REPL session useful output has been cut and pasted into this article.

This is a really basic introduction into GP and is meant for people interested in GP and/or interested in (Common) Lisp. It is heavy on code and examples and light on theory. My intention is for the reader to play around with the functions on the REPL as they come by in the article.

The code in this article should be portable Common Lisp. If you need to know what a function or macro does please consult the Common Lisp HyperSpec.

1.1 Toy Project

I found A Field Guide to Genetic Programming on-line and decided to duplicate Bill Clementson's toy project of discovering the formula for calculating the area of a circle. Except for educational purposes we will be starting from scratch.

1.2 Environment

Many Common Lispers use Emacs, Slime and usually Unix or OS X as development environment. However, do not get the impression this is the only way to write Lisp. The development page on CLiki lists many other alternatives in various states of usability.

My recommendation is to use your favorite text editor and CLISP since it comes with command line history and tab completion built in. Write or paste the code in your editor and save the file after every addition as for example "gp.lisp". Then you only need to type:

(load "/path/to/gp.lisp")

once and can use the command line history to reload it.

2 Generating Random Code

As you might have read, the space.invaders bot was written in PHP. Our weapon of choice is Common Lisp and the first matter of business is generating random code.

2.1 Operators

(defparameter *operators* '(+ - * /))

To keep things simple and already knowing our target function (πr²) we opt for using four operators: addition, subtraction, multiplication and division. In the name of simplicity (again) they will have an arity of 2, so they will all take two arguments.

*OPERATORS* is the name of the variable. The *earmuffs* convention is used to signify global variables.

2.2 RANDOM-ELT

(defun random-elt (sequence)
  (let ((seq-length (length sequence)))
    (when (> seq-length 0)
      (elt sequence (random seq-length)))))

The function RANDOM-FORM in the next section uses RANDOM-ELT to select a random element from the list of operators.

2.3 Generating Random Function Forms

(defun random-form (operators)
  (append (list (random-elt operators))
          (loop repeat 2  ; arity
                collect (let ((random-nr (random 100)))
                          (cond ((< random-nr 50) (random-form operators))
                                ((< random-nr 75) (random 10.0))
                                (t                '=input=))))))

The function works as follows: it creates a list of three items. The first item is a random operator and the second and third items are the arguments to that operator:

(operator argument-1 argument-2)

The arguments are either a random number between 0.0 and 10.0 (arbitrary limit), a variable called =INPUT= (more about this later) or, and this is important, another list of three items. For example:

(* =INPUT= 2.345)

(+ 6.789 (- =INPUT= 0.123))

So the RANDOM-FORM function will call itself recursively if needed. The (arbitrary) probabilities for the ARGUMENT-1 and ARGUMENT-2 are: 50% of the time another random form, 25% of the time a random number and 25% of the time the =INPUT= variable.

The process of generating random forms described above is similar to the "grow" method described in Chapter 2.2 of the Field Guide.

2.3.1 Limiting RANDOM-FORM

(defun random-form (operators &optional (max-depth 4))
  (append (list (random-elt operators))
          (loop repeat 2  ; arity
                collect (let ((random-nr (random 100)))
                          (if (> max-depth 0)
                              (cond ((< random-nr 50)
                                     (random-form operators (- max-depth 1)))
                                    ((< random-nr 75) (random 10.0))
                                    (t                '=input=))
                              (cond ((< random-nr 50) (random 10.0))
                                    (t                '=input=)))))))

Since RANDOM-FORM can call itself recursively it can potentially create huge amounts of random code and even exhaust the stack of the Lisp implementation. Hence we need to place some limits on RANDOM-FORM. The addition of the MAX-DEPTH parameter and the deduction of MAX-DEPTH when RANDOM-FORM is called again prevents nasty things from happening.

2.4 `=INPUT=`

Since we want our generated code to calculate the area of a circle from an input value, the radius, we've added the possibility to generate the =INPUT= variable to RANDOM-FORM. This variable is the same as the input argument of the RUN-FORM function that is discussed later.

2.5 Testing RANDOM-ELT and RANDOM-FORM

Pasting RANDOM-ELT and RANDOM-FORM into the REPL and running the latter a couple of times will yield some randomly generated code:

CL-USER> (random-form *operators*)
(/ 2.8173013 5.378826)

CL-USER> (random-form *operators*)
(* =INPUT= =INPUT=)

CL-USER> (random-form *operators*)
(+ 7.9595613
 (- (- =INPUT= (* =INPUT= 0.57189345))
  (* (- (/ 6.9174767 9.723027) =INPUT=) (* =INPUT= =INPUT=))))

CL-USER> (random-form *operators*)
(- (- =INPUT= (- (/ 2.002045 (* =INPUT= 9.829036)) 4.531122)) =INPUT=)

3 Running Generated Code

(defun run-form (form input)
  (funcall (eval `(lambda (=input=) ,form))  ; note: backquote!
           input))

To run the generated forms we wrap them in a LAMBDA with an =INPUT= argument and funcall the evaluated lambda with an input value. The following REPL interaction shows line by line what happens in RANDOM-FORM:

CL-USER> (defparameter random-form (random-form *operators*))
RANDOM-FORM

CL-USER> random-form
(* 6.341989 =INPUT=)

CL-USER> `(lambda (=input=) ,random-form)
(LAMBDA (=INPUT=) (* 6.341989 =INPUT=))

CL-USER> (eval `(lambda (=input=) ,random-form))
#<FUNCTION (LAMBDA (=INPUT=)) {AF2F08D}>

CL-USER> (funcall (eval `(lambda (=input=) ,random-form)) 2)
12.683978

CL-USER> (* 6.341989 2)
12.683978

However, the generated code can be illegal and cause errors. Since we'll be running a great many pieces of generated code multiple times we don't want to see error messages and drop into the debugger. To keep things simple we let illegal forms return NIL so we can kill them off later. Adding an error handler to RUN-FORM will do this:

(defun run-form (form input)
  (let ((*error-output* (make-broadcast-stream)))
    (handler-case (funcall (eval `(lambda (=input=) ,form))
                           input)
      (error () nil))))

*ERROR-OUTPUT* is redirected so we won't be bothered by all kinds of compilation notices. They only serve to distract in this situation.

3.1 SBCL note

SBCL is a very good, open-source CL implementation but it doesn't compile code very fast. The way we currently handle RUN-FORM by recompiling the form every time causes the advancing of generations later in the article to be very slow . Compared to SBCL, which compiles to native code, CLISP, which compiles to byte-code, does the advancing of generations much faster.

Since we're in exploration mode we will leave RUN-FORM as it is.

4 Population

(defun create-initial-population (operators &optional (size 100))
  (loop repeat size
        collect (random-form operators)))

If you're not familiar with CL: the COLLECT statement in LOOP accumulates a random form on each iteration and returns them as a list once iteration has finished. For example:

CL-USER> (create-initial-population *operators* 10)
((* =INPUT= 6.8947406)
 (* (+ =INPUT= 2.1140647) 6.7930226)
 (+
  (/ (+ (* =INPUT= (/ 8.296512 =INPUT=)) 1.1989254)
   (- =INPUT= (/ =INPUT= (- 5.1625586 1.4763731))))
  =INPUT=)
 (- 1.5379852 2.1223724)
 (* =INPUT= (- 3.0632048 (* (+ (- 5.8116364 =INPUT=) 0.1915878) =INPUT=)))
 (/ 8.237898 =INPUT=)
 (* (/ (/ (- =INPUT= =INPUT=) 0.56726336) =INPUT=) =INPUT=)
 (+ (+ 7.147339 =INPUT=) =INPUT=)
 (- (- =INPUT= =INPUT=) =INPUT=)
 (+ (/ =INPUT= (- (* =INPUT= =INPUT=) (* (- =INPUT= =INPUT=) =INPUT=)))
  6.520609))

The way we're creating the initial population is not ideal. This is because RANDOM-FORM only creates one type of syntax tree (using the "grow" method mentioned earlier). For a more diverse initial population we'd need more code generation functions that would output different kinds of syntax trees.

The "full" method is easy to add and is left as an exercise to the reader. See the Field Guide for a description.

5 Fitness

(defun fitness (form fitness-fn test-input)
  (loop for input in test-input
        for output = (run-form form input)
        for target = (funcall fitness-fn input)
        for difference = (when output (abs (- target output)))
        for fitness = (when output (/ 1.0 (+ 1 difference)))
        when (null output) do (return-from fitness nil)
        collect fitness into fitness-values
        finally (return (reduce #'* fitness-values))))

We need a way to test the output of a generated function against our desired outcome and represent this as a number: the fitness. This is commonly a number between 0.0 and 1.0. The closer to 1.0 the better the fitness.

We also need to check whether the return value of RUN-FORM is NIL in which case it executed illegal code. FITNESS will return NIL in that case as well. (Poor man's exception handling but suffices for now.)

So the function needs: 1) the form to check, 2) the fitness function to check against and 3) test input. We also want to check against multiple input values.

The TEST-INPUT argument is a list of input values so the most direct approach will be to iterate over these values and running both the form and the fitness functions against them.

To get a fitness value between 0.0 and 1.0 we take the absolute difference between the output of the generated form (OUTPUT) and the output of the fitness function (TARGET). We then add 1 to this difference and use that to divide 1.0. To illustrate:

CL-USER> (defun fit (x) (/ 1.0 (+ 1 x)))
FIT
CL-USER> (fit 0)  ; no difference so the desired output
1.0
CL-USER> (fit 0.1)
0.9090909
CL-USER> (fit 5)
0.16666667
CL-USER> (fit 500)
0.001996008

We're not testing negative values since DIFFERENCE in FITNESS will never be negative.

REPL test of FITNESS:

CL-USER> (defparameter random-form (random-form *operators*))
RANDOM-FORM

CL-USER> random-form
(/ 8.552494 =INPUT=)
CL-USER> (fitness random-form (lambda (r) (* pi r r)) '(0 1 -2))
NIL  ; this is correct since we divided by zero

CL-USER> (setf random-form (random-form *operators*))
(- =INPUT= 5.246996)
CL-USER> (fitness random-form (lambda (r) (* pi r r)) '(0 1 -2))
9.168484389517398d-4

CL-USER> (setf random-form (random-form *operators*))
(* =INPUT= (+ (* =INPUT= 1.8322039) (- =INPUT= 6.812643)))
CL-USER> (fitness random-form (lambda (r) (* pi r r)) '(0 1 -2))
0.009196622001630076d0

CL-USER> (fitness '(* pi =input= =input=) (lambda (r) (* pi r r)) '(0 1 -2))
1.0d0
CL-USER> (fitness '(* (- pi 0.1) =input= =input=) (lambda (r) (* pi r r)) '(0 1 -2))
0.649350645706412d0

6 Generation Functions

Now everything is in place to start thinking about creating new generations from the initial population. Currently we will only support cross-overs and mutations.

6.1 Traversing Nodes

Before we get to cross-overs and mutations we need to look at three important functions: N-NODES, RANDOM-NODE and REPLACE-NODE. They all share a common, but for each slightly different, function: TRAVERSE-NODES.

It is a basic recursive function. It iterates through each element of the FORM list and if that element is a list as well it calls TRAVERSE-NODES again with that list element as argument.

(defun traverse-nodes-example (form)
  (labels ((traverse-nodes (subform &optional (indent ""))
             (loop for node in subform
                   do (format t "~D:~A ~S~%" (/ (length indent) 2) indent node)
                      (when (listp node)
                        (traverse-nodes node
                                        (concatenate 'string indent "  "))))))
    (traverse-nodes form)))

TRAVERSE-NODES-EXAMPLE goes through FORM exactly how TRAVERSE-NODES does and for each node it prints its nesting level and the node itself:

CL-USER> (traverse-nodes-example '(a (b c) (d (e f) g) h))
0: A
0: (B C)
1:   B
1:   C
0: (D (E F) G)
1:   D
1:   (E F)
2:     E
2:     F
1:   G
0: H

6.1.1 N-NODES

(defun n-nodes (form)
  (let ((nodes 1))
    (labels ((traverse-nodes (subform)
               (loop for node in subform
                     do (incf nodes)
                        (when (listp node)
                          (traverse-nodes node)))))
      (traverse-nodes form))
    nodes))

Helper function for RANDOM-NODE. Returns the number of nodes in FORM including the root node. Note that "(B C)" as well as B and C are counted as nodes:

CL-USER> (n-nodes '(b c))
3
CL-USER> (n-nodes '(a (b c) (d (e f) g) h))
12
CL-USER> (n-nodes '())
1

6.1.2 Picking Random Nodes

We want to be able to pick a random node from a form to perform operations on. RANDOM-NODE does this:

(defun random-node (form)
  (let* ((index 1)
         (nodes-1 (- (n-nodes form) 1))
         (random-node-index (+ (random nodes-1) 1)))
    (labels ((traverse-nodes (subform)
               (loop for node in subform
                     do (when (= index random-node-index)
                          (return-from random-node
                                       (list :index index :node node)))
                        (incf index)
                        (when (listp node)
                          (traverse-nodes node)))))
      (traverse-nodes form))))

It picks a RANDOM-NODE-INDEX and starts traversing the nodes of FORM. When it arrives at the index it returns both the node and the index as a property list:

(:index random-node-index :node random-node)

Will not return the root node at index 0.

6.1.3 Replacing Nodes

We need to be able to replace a node in a form with another node. The avoid bugs and confusion we'll let REPLACE-NODE return this result as a new form.

(defun replace-node (form node-index new-node)
  (let ((index 0))
    (labels ((traverse-nodes (subform)
               (loop for node in subform
                     do (incf index)
                     when (= index node-index)
                       collect new-node
                     when (and (/= index node-index)
                               (not (listp node)))
                       collect node
                     when (and (/= index node-index)
                               (listp node))
                       collect (traverse-nodes node))))
      (traverse-nodes form))))

Traverses the nodes of FORM and collects them to return as a new form. When its INDEX counter is equal to NODE-INDEX it collects NEW-NODE instead.

The function does not replace the root node (index 0).

6.2 Cross-overs

There are different kinds of cross-overs but we will be doing the most straight-forward one: replace a random node in form A with a random node from form B.

Cross-overs are the most used genetic operation in GP and some people have even suggested to never use mutations. They can be compared to the procreation of humans in which attributes of the male and female are represented in their offspring.

(defun cross-over (form1 form2 &key (debug nil))
  (let ((rnode1 (random-node form1))
        (rnode2 (random-node form2)))
    (when debug
      (format t "form1: ~S~%form2: ~S~%rnode1: ~S~%rnode2: ~S~%"
              form1 form2 rnode1 rnode2))
    (replace-node form1 (getf rnode1 :index) (getf rnode2 :node))))

CROSS-OVER takes two forms as arguments and returns a new form. The new form is largely similar to FORM1 but one random node is replaced by a random node from FORM2. The function RANDOM-NODE is used to pick these random nodes.

CL-USER> (cross-over '(1 (2 3) (4 (5 6) 7) 8) '(a (b c) (d (e f) g) h) :debug t)
form1: (1 (2 3) (4 (5 6) 7) 8)
form2: (A (B C) (D (E F) G) H)
rnode1: (:INDEX 1 :NODE 1)
rnode2: (:INDEX 3 :NODE B)
(B (2 3) (4 (5 6) 7) 8)

CL-USER> (cross-over '(1 (2 3) (4 (5 6) 7) 8) '(a (b c) (d (e f) g) h) :debug t)
form1: (1 (2 3) (4 (5 6) 7) 8)
form2: (A (B C) (D (E F) G) H)
rnode1: (:INDEX 3 :NODE 2)
rnode2: (:INDEX 7 :NODE (E F))
(1 ((E F) 3) (4 (5 6) 7) 8)

CL-USER> (cross-over '(1 (2 3) (4 (5 6) 7) 8) '(a (b c) (d (e f) g) h) :debug t)
form1: (1 (2 3) (4 (5 6) 7) 8)
form2: (A (B C) (D (E F) G) H)
rnode1: (:INDEX 2 :NODE (2 3))
rnode2: (:INDEX 6 :NODE D)
(1 D (4 (5 6) 7) 8)

CL-USER> (cross-over '(1 (2 3) (4 (5 6) 7) 8) '(a (b c) (d (e f) g) h) :debug t)
form1: (1 (2 3) (4 (5 6) 7) 8)
form2: (A (B C) (D (E F) G) H)
rnode1: (:INDEX 5 :NODE (4 (5 6) 7))
rnode2: (:INDEX 5 :NODE (D (E F) G))
(1 (2 3) (D (E F) G) 8)

6.3 Mutation

Mutations change a part of a form without needing another form for the operation. It is perhaps analogue to a cosmic ray changing a bit of genetic information in a biological entity.

We will be using subtree mutation since it is straight-forward to write and has a potentially large effect.

(defun mutate (form operators &key (debug nil))
  (let ((rform (random-form operators))
        (rnode (random-node form)))
    (when debug
      (format t "form: ~S~%rform: ~S~%rnode: ~S~%" form rform rnode))
    (replace-node form (getf rnode :index) rform)))

Mutation replaces a random node of FORM with a form created by RANDOM-FORM.

CL-USER> (mutate '(a (b c) (d (e f) g) h) *operators* :debug t)
form: (A (B C) (D (E F) G) H)
rform: (+ =INPUT= (- 4.4699216 (+ 9.623513 =INPUT=)))
rnode: (:INDEX 3 :NODE B)
(A ((+ =INPUT= (- 4.4699216 (+ 9.623513 =INPUT=))) C) (D (E F) G) H)

CL-USER> (mutate '(a (b c) (d (e f) g) h) *operators* :debug t)
form: (A (B C) (D (E F) G) H)
rform: (+ 4.5209084 (- 8.943897 (+ 6.657296 =INPUT=)))
rnode: (:INDEX 2 :NODE (B C))
(A (+ 4.5209084 (- 8.943897 (+ 6.657296 =INPUT=))) (D (E F) G) H)

7 Advancing a Generation

7.1 Evaluating a Population

(defun evaluate-population (population fitness-fn test-input)
  (loop for form in population
        for fitness = (fitness form fitness-fn test-input)
        when fitness collect (list :fitness fitness :form form) into result
        finally (return (sort result
                              (lambda (a b)
                                (> (getf a :fitness) (getf b :fitness)))))))

To advance a generation we need to do cross-overs and mutations to the forms in the population. The literature suggests we give forms with a higher fitness more chance to be a candidate for a cross-over or mutation.

EVALUATE-POPULATION evaluates a population and returns a list of the forms in that population and their fitness (minus any forms that gave errors). For ease of testing on the REPL and for use in our next function the output is sorted from best fitness to worst:

CL-USER> (defparameter population (create-initial-population *operators* 10))
POPULATION

CL-USER> (evaluate-population population (lambda (r) (* pi r r)) '(0 1 -2))
((:FITNESS 0.016815323605722716111d0 :FORM (- (- =INPUT= =INPUT=) =INPUT=))
 (:FITNESS 0.009437720627636827027d0 :FORM (- 1.5379852 2.1223724))
 (:FITNESS 0.007690745276452599039d0 :FORM (* =INPUT= 6.8947406))
 (:FITNESS 0.0031801166742824690382d0 :FORM
  (* =INPUT= (- 3.0632048 (* (+ (- 5.8116364 =INPUT=) 0.1915878) =INPUT=))))
 (:FITNESS 0.0016815216288940106286d0 :FORM (+ (+ 7.147339 =INPUT=) =INPUT=))
 (:FITNESS 2.6768631466526024146d-4 :FORM (* (+ =INPUT= 2.1140647) 6.7930226)))

This is the same population as created in the Population chapter. Note that four of the ten forms have been eliminated since they executed illegal code.

7.2 HEAD

(defun head (sequence &optional (amount 1))
  (if (<= amount 0)
      nil
      (if (< (length sequence) amount)
          sequence
          (subseq sequence 0 amount))))

Utility function used by ADVANCE-GENERATION. Returns AMOUNT elements from the start of SEQUENCE. If SEQUENCE is shorter than AMOUNT it will return the whole SEQUENCE.

7.3 Running the Advancement

(defun advance-generation (population fitness-fn operators test-input
                           &optional (max-population 100))
  (let ((epop (evaluate-population population fitness-fn test-input)))
    (format t "Best fitness of current population: ~S~%"
            (getf (first epop) :fitness))
    (loop for plist in (head epop max-population)
          for i from 0
          for fitness = (getf plist :fitness)
          for form = (getf plist :form)
          collect form
          when (<= (random 1.0d0) fitness)
            collect (if (<= (random 100) 90)
                        (cross-over form (getf (random-elt epop) :form))
                        (mutate form operators))
          ;; Add a new random form to the population now and then.
          when (<= (random 100) 2) collect (random-form operators))))

Using the list of forms and their fitness from EVALUATE-POPULATION as input we only loop over the first MAX-POPULATION items to keep the population size in check. We then check the form's fitness against a random number between 0.0 and 1.0 and if the number is lower than the fitness the form will be selected for either a cross-over (90% chance) or a mutation (10% chance).

So the form's fitness is its chance to be selected. Since the fitness of the forms in the initial population is usually very low it will take quite a few generations before something starts happening.

We collect the form and if it's been selected we also collect the result from either CROSS-OVER or MUTATE. These will all be accumulated and eventually returned as the new generation.

There's also a small chance a new random form will be added to the population since the way we're currently handling things it is very possible for a successful form to take over the entire population and make it too homogeneous for good results. This is a quick and easy hack to introduce some chaos into populations.

7.4 Finding a Solution

Lets run ADVANCE-GENERATION a hundred times over a new population (we also update the population on each iteration with SETF):

CL-USER> (defparameter population (create-initial-population *operators* 100))
POPULATION
CL-USER> (loop repeat 100
               for i from 0
               do (format t "[~S] " i)
                  (setf population
                        (advance-generation population
                                            (lambda (r) (* pi r r))
                                            *operators*
                                            '(0 1 -2))))
[0] Best fitness of current population: 0.08546627574466652d0
[...]
[43] Best fitness of current population: 0.08626213422506805d0
[...]
[66] Best fitness of current population: 0.10050107335294403d0
[...]

And again:

CL-USER> (loop repeat 100 for i from 0 do (format t "[~S] " i) (setf population (advance-generation population (lambda (r) (* pi r r)) *operators* '(0 1 -2))))
[...]
[26] Best fitness of current population: 0.3865141184760903d0
[...]
[93] Best fitness of current population: 0.49884414719455095d0
[...]

Lets do another 300 runs so we've done 500 in total and see what the best form looks like in the end:

CL-USER> (loop repeat 300 for i from 0 do (format t "[~S] " i) (setf population (advance-generation population (lambda (r) (* pi r r)) *operators* '(0 1 -2))))
[...]
[35] Best fitness of current population: 0.5397727425319018d0
[...]
[62] Best fitness of current population: 0.559234953025742d0
[...]
[117] Best fitness of current population: 0.6436990901489741d0
[...]
[179] Best fitness of current population: 0.9657338407311192d0
[...]
[201] Best fitness of current population: 0.9705968409735506d0
[202] Best fitness of current population: 0.9755729636966523d0
[203] Best fitness of current population: 0.9994359703065686d0
[...]

CL-USER> (defparameter best-form (first (evaluate-population population (lambda (r) (* pi r r)) '(0 1 -2))))
BEST-FORM
CL-USER> best-form
(:FITNESS 0.9994359703065686d0 :FORM
 (* (+ (+ =INPUT= =INPUT=) (+ =INPUT= (/ =INPUT= (+ 3.5050452 3.5518444))))
    =INPUT=))

CL-USER> (run-form (getf best-form :form) 0)
0.0
CL-USER> (run-form (getf best-form :form) 1)
3.1417055
CL-USER> (run-form (getf best-form :form) 2)
12.566822
CL-USER> (run-form (getf best-form :form) 3)
28.275349

Here's the output from the real function to calculate the area of a circle for comparison:

CL-USER> (run-form '(* pi =input= =input=) 0)
0.0d0
CL-USER> (run-form '(* pi =input= =input=) 1)
3.141592653589793d0
CL-USER> (run-form '(* pi =input= =input=) 2)
12.566370614359172d0
CL-USER> (run-form '(* pi =input= =input=) 3)
28.274333882308138d0

Not quite the impressive result of Bill Clementson's attempt but not half bad either!

8 Conclusion

I hope to have shown how one can start with a few simple functions (RANDOM-FORM and its RANDOM-ELT helper) and explore a topic which might interest you.

If you read the literature you'll notice that there are a lot of things still wrong with the current setup. As you will find out if you start experimenting with the code in this article bloat is a big problem, as well as a successful form taking over the entire population.

If the topic of GP interests you I would suggest you read up and look into:

eliminating bloat
improving the creation of the initial population
improvements to the cross-over and mutation functions

for the code in this article.

9 Thanks

Thanks to the denizens of #lisp at freenode for Common Lisp help in general.

Thanks to the following people for proofreading and comments: Marijn Haverbeke, Gábor Melis, David O'Toole and Matthias of the space.invaders team.

Keep in mind that any mistakes are mine.

Planet Wars: You Can Still Participate

2010-10-11T08:36:00.000+02:00

The Planet Wars competition has been running for a month now and the deadline for submissions is Saturday the 27th of November 2010.

Don't let the fact that it has already been running for a month dissuade you from participating. If you're an experienced programmer a weekend of solid work will get your bot in the top 500 (out of a little more than 3000 participants currently) and maybe even in the top 100. If you're less experienced perhaps a week or two of work will get you the same (and if not, you will still learn a lot!).

Some perspective: my bot briefly entered the top 25 this weekend (but it's on its way out of the top 50 currently) and I've been working on it in my spare time the last two or three weeks. I'm not experienced in AI programming and I've got a full time job, a wife and two young kids to distract me.

So, grab a Common Lisp starter pack and get hacking!

One tip: The map format is very simple (check the maps directory in any starter pack). Make simple maps to test your attack and defense routines on.

Some helpful links:

dhartmei's TCP server: with this you can play games against others from your own machine (ie. without uploading) and it gives you more games in a shorter amount of time than the official server.
JBotManager: I use this utility exclusively for replaying games.
Test scripts for running your bot against the example bots on all the maps. If you win against these bots 95% of the time you'll end up in the top 500 and if you win against them 100% of the time you'll probably end up in the top 100.
A strategy guide with links to other strategy guides.
Parsing the game output state: this will allow you to get a played game back from the official server in a way that can be fed back into your bot. This has helped me find some bugs of games where my bot crashed on the official server which I missed when testing locally.

Planet Wars: Common Lisp Starter Package, part 2

2010-09-21T15:40:00.000+02:00

It took a while but Common Lisp is finally supported on the official Planets Wars game server (Google AI Challenge)! (In the meantime one could use dhartmei's server (and you still can since it offers some advantages like more games in a shorter time).)

Both Gábor Melis and I updated our starter packages which can be found here:

My starter package has been improved a little since earlier versions (most importantly no reliance on *features* anymore) and it comes with a silly bot that is currently slowly climbing the ranks: http://ai-contest.com/profile.php?user_id=4055

Planet Wars: Common Lisp Starter Package (Google AI Challenge)

2010-09-09T23:01:00.006+02:00

The Google AI Challenge should start any day now and I and "anwyn" have both made Common Lisp starter packages. anwyn's follows the API of the other starter packages more and is perhaps a little more lispier (I like LOOP a lot) and mine's a little more bare-bones: just the basic communication with the game server and a silly example and you have to do the rest.

One little extra in my starter package is a "proxy bot". Since the initial starter packages use conventional languages, developing and running the bot is also pretty conventional: a basic edit, compile, run (crash, coredump) loop. The proxy bot takes it on himself to be the one that's started up and shut down every time[1] instead of your real bot, so you can keep the latter running in an, in my case, Emacs + Slime session.

(I don't have to tell the advantages of this to this audience.)

This proxy bot is perhaps useful to people writing their bots in Clojure and Scheme (and Haskell?) as well. Although they will have to adapt their main loop a little to allow for socket communications besides the normal way of talking to the game server.

Also, there's an unofficial server on which you can test your bot against others: http://www.benzedrine.cx/planetwars/

[1] You don't really have a choice with the Planet Wars game engine.

New Google AI Challenge: Planet Wars

2010-09-06T10:28:00.000+02:00

There's a new Google AI Challenge: Planet Wars.

It seems they're also working on adding support for Common Lisp due to the number of people that used it in the last challenge. Awesome!

ILGE 2010 Post-mortem: Engine Troubles over Tentacle Planet

2010-08-09T13:41:00.000+02:00

Progress

I have made very little progress since the last report, mainly due to the little time that was available to me. Only two things were done:

The source code was cleaned up a little.
A windows binary was made (OpenGL only).

In the short term I will not work on ETOTP anymore but perhaps for ILGE 2011 or another challenge or expo I'll continue where I left off.

What Went Right

Making wrapper functions in C for Ogre's C++ methods and adding them to ECL from C using ECL's API. I used this approach because the inverse (making the C wrapper code from Lisp) gave me segmentation faults and didn't get me very far about 2 years ago. (Note: this was using ECL functionality, not CFFI for which there's Okra.)
Compiling a C program on both Linux and Windows that was (dynamically) linked to ECL, Ogre and OIS. This C program can, for example, start a game right away and quit back to the OS but it can also provide a CL REPL (like ECL's si:toplevel) so one can interactively play with a 3D scene. Since it's just a plain CL one can load extra packages and perhaps start Slime.

What Went Wrong

Compiling a statically linked binary on Linux. This didn't go as easily as I had hoped and was starting to eat into my time budget.
ManualObjects and shadows. I had to resort to Ogre's prefab entities to get shadows working the way I wanted in the time that I had available.

What I Learned

ECL is a really good option for embedding a CL implementation in C(++) programs. Juanjo has been working on ECL for years now and he's still going strong. Questions on the mailing-list are generally answered within hours.
One doesn't just add "-static" to a build rule and be done with it :-)
Perlin noise is usable to animate objects with. Definitely for prototyping since it's so easy and quick to implement.
I should brush up on my maths.

Source & Binary

The source code and the Windows binary are available here
http://www.aerique.net/software/etotp/.

I have not given the source code its separate GitHub repository since it will be added to Okra.

Linux Dependencies

ECL (10.4.1) configured "–with-cxx". (Feel free to try without it, I haven't.)

Debian: libogre-dev libois-dev ogre-plugins-cgprogrammanager

Windows Dependencies

I develop on Linux but the build is checked and tested on Windows using MinGW and MSYS. Either MSYS 1.10 or 1.11 was used, it doesn't really matter. The MinGW release that was used is TDM-GCC 4.4.1 but I see there's a 4.5.0 release out now.

Ogre SDK 1.6.1 for MinGW (there's no later 1.6.x SDK for MinGW). You can try the 1.7.x SDK for MinGW since the Ogre API is pretty stable, but I can make no guarantees that it will work.

ILGE 2010: Engine Troubles over Tentacle Planet, part 2

2010-07-22T22:31:00.001+02:00

1 Progress

Since the last report I've added: a controllable spaceship, bullets, tentacles and I made an unsuccessful attempt at producing a Linux binary.

For the controls I use the OIS library. I was already familiar with this due to my clois-lane library. The controls work but don't feel quite right yet and need some minor adjustments.

Adding bullets was straightforward. The most time was put into the tentacles and the attempt at making a statically linked Linux binary.

The tentacles are Ogre prefab cubes plot along a Bézier curve. The P1 and P2 control points and the P3 end point are assigned a new x,y,z coordinate each frame. This coordinate is picked by walking through 3D Perlin noise with a random movement vector for each point. (The vector was created randomly at creation time, it's stays the same from frame to frame.)

The code is a huge mess by now and it is getting in the way. Next priority should be a clean-up.

2 To Do

[X] rolling terrain (high)
[X] controllable spaceship (high)
[X] improve controls (low): The controls aren't quite there yet,especially moving left and right.
[X] bullets (high)
[X] cmake file (low)
[X] basic tentacles (high)
[X] shootable tentacles (high)
[X] collision detection: spaceship vs terrain (high)
[X] collision detection: spaceship vs all other entities (high)
[X] tentacles that shoot small spores (medium)
[X] roaming (bigger) spores that attack the player (medium)
[X] some minor GUI elements showing the score, highscore and FPS. (high)
[X] make binary release for Windows (medium)
[X] spaceship needs to cast a shadow in the terrain to judge heights(high): Solved this by using prefab entities instead of a manual object.
canceled: make binary release for Linux (high): This is just too much of a hassle. The Ogre package for Debian didn't come with static libraries and compiling it myself with "–disable-shared –enable-static" didn't produce them either so this is going to take too much time.
Eventually I managed to build static Ogre libraries but ran into other problems when compiling with "-static". Statically compiling is something I have no experience with and this looks like it is going to take too much time for the ILGE 2010.
For Linux it seems distribution by deb and rpm packages will be the best solution. (What a revelation!)

3 Source

The code for the latest version isn't available yet since it is a horrible mess. Keep watching http://www.aerique.net/software/etotp/ (not up yet) if you're interested.

3.1 Linux Dependencies

ECL (10.4.1) configured "–with-cxx". (Feel free to try without it, I haven't.)

Debian: libogre-dev libois-dev ogre-plugins-cgprogrammanager

ILGE 2010: Engine Troubles over Tentacle Planet, part 1

2010-07-13T12:28:00.003+02:00

Introduction

I've finally managed to scramble together a good couple of hours to work on my ILGE 2010 entry. At this rate it will be doubtful whether I'll have a minimal game by the end of this month :-(

The first item that can be checked off the to-do list: rolling terrain. The video is low quality and stutters but this is due to me using sub-optimal software to capture my desktop. The program actually runs very smoothly on both systems I've ran it on (a Samsung NC10 netbook and an Intel P4 2.8Ghz desktop).

The video shows the planet on which this game is played. It is a standard heightfield created using Perlin Noise (with the CL Black Tie library) which keeps scrolling down endlessly until the player dies.

Goal

Explore the usability of Ogre and C++ from Embeddable Common Lisp (ECL) using functionality specific to ECL as opposed to using the CFFI approach used by Okra. (Since Okra's CFFI approach needs to compile a small C wrapper library to access Ogre's C++ functions anyway.)

I will be ignoring any licensing issues that might come with releasing a binary that's statically linked to ECL and Ogre for now.

Development Environment

My development environment is Debian Linux. Testing will also be done on Windows (Vista unfortunately) in the future but that hasn't happened yet.

Dependencies

ECL 10.4.1
Ogre 1.6.4
OIS 1.2.0 (not used yet)

ECL has been compiled by myself but Ogre and OIS have been installed using Debian's package manager (you need libogre-dev and libois-dev).

I have not used Ogre 1.7.x yet but since Ogre's API has been pretty stable in the past it might just work without any changes.

TO-DO

In chronological order:

[X] rolling terrain
[X] controllable spaceship
[X] basic tentacles
[X] bullets
[X] shootable tentacles
[X] tentacles that shoot small spores
[X] roaming (bigger) spores that attack the player
[X] collision detection: spaceship vs terrain
[X] collision detection: spaceship vs all other entities

Don't know when:

[X] some minor GUI elements showing the score, highscore and FPS.

Source

The code is not pretty but available here: http://www.aerique.net/software/etotp/etotp-20100713.tar.gz

I'm slowly moving from using global variables to stuffing everything into a scene-class instance so that will account for any duplication you might see.

Common Lisp source to Markdown converter

2010-03-17T12:10:00.001+01:00

cl2md is a quick hack inspired by pbook.el (which you should check out if you've never heard of it and are an Emacs user since it marks up the source code much nicer).

The page "cl2md" in the previous paragraph links to is the actual source code file which has been run through itself and a Markdown converter.

StarCraft ProxyBot Client for Common Lisp

2010-03-10T14:12:00.000+01:00

Sorry for the CamelCase but I didn't come up with the names :-)

For those bitten by the game AI bug since the Google AI Challenge (all eleven of us) I've started working on a Common Lisp (CL) client for the StarCraft ProxyBot with the StarCraft AI competition in mind: http://github.com/aerique/cl-starcraft-proxybot.

The project is still in its early stages but the important groundwork is done: commands can be send to in-game units.

Before this project I started working on a CL client for GameBots 2004 with which one can control an Unreal Tournament 2004 avatar. The project's stalled for now and hasn't come much farther than setting up the initial connection but for those interested it is here: http://www.aerique.net/software/.

Lastly for those interested in game AI: TAEB is on my radar. See also this excellent blog post: Planar - a TAEB AI.

Google AI Challenge 2010 part 2

2010-02-10T15:10:00.002+01:00

They've finally added Common Lisp as a supported language to the Google AI Challenge. (Just before they stopped adding new languages, phew!)

There were some issues getting my starter pack to work on their server since they're running an older SBCL (1.0.18 on Debian) and that didn't support the "--script" switch. That combined with them being very busy meant it took a while. We were passing error messages and solution back and forth only once every other day.

So now the code is on the server side is compiled (#'save-lisp-and-die with :executable t) which means you can make good use of the two extra seconds you get for the first move (which some other languages need for starting up). Anyway, check the Makefile and example.sh for details.

There's still some issues with the starter pack if you're running locally and you don't want to run "make" for every other change (and ofcourse you don't want that!) so check the "Common Lisp pack?" thread in their forums for discussion and solutions.

Google AI Challenge 2010

2010-02-05T16:43:00.003+01:00

For those who've missed it: Google AI Challenge 2010

There's no Common Lisp starter pack (edit: since this is getting misinterpreted I meant a CL starter pack for the AI Challenge) but after a particular nasty adventure spelunking for days through an archaic database at work, fighting through the 20 years of cruft that has been build up inside of it and finally finding and fixing the problem I decided that I needed a break and wrote an initial version of a Common Lisp (SBCL) client.

Perhaps someone with connections at the University of Waterloo and some work can finish it and make it an official starter pack? I've got to go back to work ;-)

Relevant links:

A Black Tie and a Clysma

2010-02-01T10:41:00.003+01:00

I've uploaded one-and-a-half project to GitHub that may or may not be useful to other people:

Black Tie packages the noise functions that were previously in Okra as a seperate library. It is not very extensive yet and has only been optimized for SBCL. At the moment it includes functions for generating Perlin noise, simplex noise and Voronoi diagrams.

Clysma is a toy project that at the moment mostly serves as an example of using CL-GTK2. I don't really know where I want to go with it but for now it allows you to stroll through a running CL session and do some sightseeing.

De Nederlandse Politiek van Tegenwoordig

2010-01-13T10:40:00.001+01:00

Vroeger trad er nog wel eens een minister af als deze over de scheef was gegaan, maar tegenwoordig trekken ze een Rita Verdonk-hoofd, doen alsof feiten meningen zijn, zeggen dat ze tóch gelijk hebben en gaan gewoon lekker verder.

Easy Okra on Debian

2009-12-25T21:23:00.005+01:00

I was installing Debian (testing release) on a netbook since my main laptop is away for repairs and I noticed the Ogre packages in the distribution. So far I have always compiled Ogre and its dependencies myself so I was curious how much work it would take to get the Okra examples running.

Not much work, but you do need to pull the latest master branch from GitHub since I had to make a few corrections.

To run simple-okra.sh you'll need to install the following packages: cmake, g++, libogre-dev, libogremain-1.6.4 and ogre-plugins-cgprogrammanager. Go to Okra's root directory, turn BUILD_CEGUI off in CMakeLists.txt, run "cmake -G 'Unix Makefiles'" and finally run make. Then go to the examples-directory and run ./simple-okra.sh.

To run flock.sh you'll need OIS and CEGUI as well. The latter pulls in a lot of extra dependencies (which is why I'll be switching to MyGUI). Install these additional packages: libcegui-mk2-1, libcegui-mk2-dev, libceguiogre-dev, libceguiogrerenderer-1.6.4, libois-1.2.0 and libois-dev. (You'll also need to get clois-lane from GitHub, run cmake and make for it and copy lib/libclois-lane.so to okra/lib.) Turn on BUILD_CEGUI again if you turned it off above, run cmake and make again and then run ./flock.sh in the examples directory.

I didn't try running physics-and-input.sh since Bullet isn't available as a package in Debian/testing. It only needs OIS and not CEGUI if I remember correctly.

Embedding an Ogre render window in GTK using Okra

2009-12-05T00:22:00.009+01:00

Thanks to CL-GTK2's author Dmitry Kalyanov for his help and patience and also thanks to several #lisp residents for testing some of my builds.

Go to http://www.aerique.net/software/okra-gtk-demo/ for the downloads.

This post should be treated more as a work-in-progress report¹ than as the blog I hoped I could publish, which would have had a title like: "Easy cross-platform executable delivery using CL-GTK2 and Okra".

So this post should be considered as a first step and perhaps a helpful reference for people with the same intent as I did. It should definitely not be treated as a completed project or as a reference on the proper way to build such things in Common Lisp (CL)!

I set out to write a little demo that would have an Ogre window embedded in a GUI application and that could distributed as an executable on both Linux and Windows. (I have neither access to nor experience with OS X.)

I settled on CL-GTK2 for the GUI². I also wanted it to work on SBCL for Windows so that ruled out using threads. Not a smart decision on my part since it would have been better to use a separate thread for the GUI. Eventually I didn't use SBCL on Windows anyway since it didn't work out how I initially used CL-GTK2. (It might work using #'GTK:GTK-MAIN but I never tried.)

Keep in mind I used Glade to construct my GUI. CL-GTK2 comes with many examples if you want to construct your GUI programmatically.

Things to be aware of when embedding an Ogre window in GTK

Let CL-GTK2 run in its separate background thread, this will avoid many awkward constructs and unresponsiveness in the GUI.

Use a GTKEventBox for embedding the Ogre window if you need to catch events. It also needs to be declared "can focus" and of course you'll need to enable all the events you want to catch.

Ogre: You'll need to use Ogre::WindowEventUtilities.messagePump() in configure_event or the GUI will hang when it is being realized.

Linux: I had better success using the downloadable 0.1 release than the latest from the Git repository. Especially with signals the GUI just hung. I'll still advice you to use the latest version from the repository but if you're running into these kind of problems you'll know what to try. When using the 0.1 release gtk/gtk.widget.lisp will need to be patched: http://github.com/dmitryvk/cl-gtk2/commit/804b4c8f24b3725eb90f29d7e6910b2598b68771 (%gtk-widget:state and -:saved-state need to become :uint8).

Windows: I couldn't restart a Clozure CL (CCL) saved application when using the CL-GTK2 0.1 release. However the latest from Git HEAD also had issues which were resolved by commenting out (:file "gdk.threads") in gdk/cl-gtk2-gdk.asd. See: http://github.com/dmitryvk/cl-gtk2/commit/061cf59f4abaa71f92182ee8db56e023208718d0

Windows: I had to declare the event box as "above child" for the mouse events to be caught.

Windows: The Ogre render window needs to be repositioned to 0,0. At the start it will be offset by a couple of hundred pixels.

Unresolved issues

CCL: To be fair I didn't put in the time to figure this out but I'm not sure what the top-level function given as argument to SAVE-APPLICATION needs to work like. I wasn't able to reload the foreign libraries using this function and I needed to use the :INIT-FILE argument.

SBCL: It doesn't seem to be possible to set cffi:*foreign-library-directories* before the saved image starts looking for them. Because of this I still need a start-up script.

CCL vs SBCL: On CCL (at least on Windows) you'll need to manually reload the foreign libraries when starting up the saved image. SBCL does this automatically.

Resizing doesn't really work well yet. Ogre::RenderWindow: update vs windowMovedOrResized vs renderOneFrame?

I wasn't able to get a configure_event to fire for the GTKEventBox nor was I able to make it available from Glade for the top level window.

I also was not able to make the destroy event for the top level window available from Glade.

How do I neatly align the text in the camera, mouse and statistics frames using Glade? I've been fooling around using different approaches but didn't succeed.

What's the easiest way to distribute an executable on Linux? :-) (don't answer this)

Ogre: When to use messagePump?

Linux: BadDrawables, I get these now and then and I've tried and tried to debug them but since they come only sporadically it's hard. What causes them? My guess would be Ogre trying to use an invalid GDK window but several different approaches to fix this were all unsuccessful.

Notes on the source-code

The source-code is in a "Hey it compiles, ship it!" state, albeit a little cleaned up. Still I think it is more useful to no source-code at all. The intention was to provide a single package that would work on both Linux and Windows, hence the init.lisp script.

I got familiar with CL-GTK2 and GTK throughout this project which explains several awkward constructs in the source-code.

The demo uses keyboard scan-codes. I know this is hopeless.

I'm depending on the keyboard repeat for continuous movement instead of the key press and release events like one should for nice control and feedback. Check the Okra demos for how that should be done.

This project was developed on Linux using SBCL 1.0.31.debian and GTK 2.18.x and on both Windows XP and Vista using Clozure CL and GTK 2.16.x.

Footnotes

¹ On which I won't be spending any more work in the near future.

² I tried out several GTK and QT bindings for CL and CL-GTK2 was the easiest to get working on both Linux and Windows of the active projects.

Torchlight Released

2009-10-28T14:22:00.003+01:00

Torchlight has been released by Runic Games. It's is a hack & slash game in the grand tradition of Rogue, Diablo and Diablo 2. Key people that worked on the last two titles have also worked on Torchlight.

The reason I blog about this is because it uses Ogre for its graphics engine which is the same library Okra uses.

I played it for a couple hours last night and it is really enjoyable. So go check it out it's only $20 and can be bought from their site, Steam or any of the other online distributors.