Michael Clarke

At last! I’ve finally started doing some work on my major project. I’ve spent a brief amount of time yesterday getting my new Trac installation working. You can get to the Trac install here, and unlike last time everything is restricted to read-only for various reasons:

• It’s supposed to all be my own work!
• Last time I put a link to a Trac install I got thousands of tickets which suggested I buy Viagra.

Anyway, having installed Trac I then set about creating some components, milestones, versions etc. I also started writing some documentation about the OS, the coding standards to be used, the goals, and all that sort of stuff. As I figure out more completely the requirements of the project the Wiki will be improved with more information and documentation. My hope is that when it comes to the final report I’ll be able to grab a lot of the stuff off the Wiki.

You may be thinking to yourself - this is all well and good, but this isn’t actually a start - it’s just a “time wasting exercise to make you think you’ve started.” But this is where you’d be wrong. In addition to all the above I created the first ticket - the boot loader task, and then promptly proceeded to write the boot loader. Although I’ve not tested it yet, it’s so simple stuff I don’t have any concerns. Obviously that won’t do when it comes to writing my report and all that palava - so I will have to come up with a way of testing a boot loader, but for the moment I’m just assuming it will work - it compiles.

As with MouthOS, the boot loader for my project uses GRUB, but this time it makes much more effort to use the features provided, such as GRUB’s own memory map etc. This is actually a very simple task:

kernel:

extern main

; Push the address of the GRUB memory info onto the stack so that
; the kernel can get to them.
push eax;
push ebx;

; Call the main kernel. Notice that we should NEVER EVER return
; from main, but if we do we end up in an infinite loop. At
; some point (once system calls are working) that should change
; to panic!
call main
jmp $

Easy eh? Well, there is actually a little more code than that in order to setup a basic 8K stack, BSS etc. However, that is the gist of the boot loader! You’ve gotta love GRUB. Obviously there are a number of things that still need doing such as enabling interrupts, setting up the Global Descriptor Table and Local Descriptor Tables etc. These are things that GRUB does, but only to a very simple extent - for example, as my comments in the code say:


; cs  = A read/execute segment, offset: 0, limit: 0xFFFFFFFF (32 bit)
;
; ds, es, fs, gs, ss
; A read/write segment, offset 0, limit: 0xFFFFFFFF (32 bit)


You can very quickly see that the entire of the memory is read/write/execute - no matter if it is a stack, code segment, data segment etc. In other words GRUB would seem to be pretty lazy in this regard. But I suppose that is because no matter what you’re going to want to change it anyway. So my next task really is to write the rest of the boot loader including the code to sort out the GDT and LDTs. Oh, and I’ve added about 25 tickets now of things that I need to do for the first release of the project. Looks like I’ve got a lot of work to be doing.

So, apart from my progress with the major project, what else have I been doing? I’ve started going swimming and last Thursday I managed to do 90 lengths (approximately two a half Km) - perhaps not fantastic for some, and a real shame that it’s not 100, but for me that is pretty good going. Today I also managed to re-create this achievement. Hopefully next time I’ll manage the 100 - but by 90 I’m exhausted!

So right now I’m off to get a cup of coffee and then I’m going to watch the latest episode of Tourchwood on iplayer, and then tomorrow I’ve got guitar lesson

At last it has happened! I have been putting off writing my memory allocator for MouthOS for some time. Up to now I’ve been able to survive without one by simply not having any ‘undefined’ data structures - everything in the kernel was a known, static size.

So after a interesting Solaris Advanced Crash Dump Analysis course last week (which has really put me back into my OS development mood) I decided it was time to make a move and I spent a large portion of Friday night working out how to allocate memory without overwriting the existing kernel, and at last by about 2am in the morning having messed around with the linker and some cleverly placed assembly labels I finally was able to kmem_alloc() some memory.

At this point I decided I was finished and went to bed. The next morning I awoke, pleased to have kmem_alloc() working, or so I thought. However, as the day progressed and I allocated more and more pages of memory I started to see some strange things happening. Why did that character on screen just go red? I never put a start on the screen…

After a fair amount of time debugging the issue I managed to isolate the problem. My array of free_pages_t structures were located, not at the end of the kernel stack as I thought (which is currently 8K), but in the last 1K of the stack. Ultimately the problem was a off by one (er, or 1024 depending on how you look at it) error in my programming. At last I was able to kmem_alloc() pretty much as much memory as  I wanted - but how to free.

This proved to be very easy. I simply modified my page_map_t structure to contain a grouping id. As such, when we call kmem_free(void *) we do the following to first make sure that the address we got was on a page boundary.

	while ( (addr % (PAGE_SIZE * 1024) != 0 )
		addr--;

	while ( page->addr != addr )
		page = page->next;

Now we’re on a page boundary we can just loop through the pages whilst the alloc_id’s are equal and set the flags to PAGE_FREE…

	int alloc_id = page->alloc_id;

	while ( page->alloc_id == alloc_id ) {
		page->alloc_id = 0;
		page->flags &= PAGE_FREE;
		page = page->next;
	}

With that working I then spent a bit more time perfecting and tweaking and now MouthOS is pretty good at allocating and freeing memory - at least, I think it is anyway. In the end it turned out not to be so difficult - you just have to get your head around the whole idea of allocating memory, before you have an allocator. When I thought about it this ended up not to be such an issue after-all….  Here is how we can do it.

1. You know the point where your kernel loads.
2. You can work out where your kernel ends by putting a label in its own section, and then using the linker to put that section at the end of the kernel.
3. You now know (from the placed label) where a safe place in the memory is to write without overwriting the existing kernel, so write all your required memory structures, of which you can calculate how many you need from the amount of memory in the system.
4. Finally,go back to your memory structures that you’ve just written and mark all the kernel space as allocated, including the space the memory structures take - which you know as you know the size of a memory struct, you know where it starts, and you know how many you’ve got.

Easy You can find the code for the new MouthOS memory allocator at: http://www.michael-clarke-blog.com/trac/MouthOS  -  if you’re remotely interested :p

Until next time.

It’s taken me some time to gather the enthusiasm to do this - but I’ve finally moved my MouthOS source over to my web server and installed trac. So, you can view the MouthOS wiki and source at:

http://www.michael-clarke-blog.com/trac/MouthOS

The older versions are in /proto. Kernel version 0.0.1, 0.0.2 and 0.0.3 are broken with some pretty big bugs (for such small programs) such as triple faults and Kernel panics.

The most stable version at the moment is version 0.0.03 in /releases.

The stuff in /trunk is the current development with all the new HAL (Hardware Abstraction Layer) stuff. I’m still in the process of porting a lot of the proto-0.0.3 and releases-0.0.3 stuff to /trunk - so this is pretty useless at the moment But you’ll get the general idea as to where the project is heading from /trunk in terms of the HAL, Memory Management etc.

At some point I’m also going to add anonymous svn access so you’ll all be able to checkout the source code and have a play - but at least for the moment you can view the code, the wiki etc.

For the moment - enjoy

The other day I wrote a small tutorial on getting dynamic class loading working for Java. In that post I mentioned that it could also be done in C# - but that it was more difficult, and from my personal experience, there is less information available about it - so here is how to do it!

First, as with the Java, we need to create our API. However, unlike the Java there is an additional requirement - a ‘module manager’ class needs to be within the API. As such you will need to create these two files…

using System;
using System.Reflection;

namespace com.michaelclarkeblog.api {

  public static class ModuleManager {

    public static Module getInstance(String fileName) {

      /* Load in the assembly. */
      Assembly moduleAssembly = Assembly.LoadFile(fileName);

      /* Get the types of classes that are in this assembly. */
      Type[] types = moduleAssembly.GetTypes();

      /* Loop through the types in the assembly until we find
       * a class that implements a Module.
       */
      foreach (Type type in types) {

        if (type.GetInterface("Module") != null) {

          /* Create a new instance of the 'Module'. */
          return (Module)Activator.CreateInstance(type);

        }
      }

      return null;
    }
  }
}

using System; 

namespace com.michaelclarke.dynamic.api {
  public interface Module {
    String getText();
  }

}

Once you’ve got the two files you can compile them into a library….

api $ mcs -t:library Module.cs ModuleManager.cs
api $ ls -a | grep .dll
Module.dll
api $

Now that we’ve made the API the next thing to write is the application that is going to use it for loading modules. This is a very simple application…

using System;
using com.michaelclarkeblog.dynamic.api;

namespace com.michaelclarkeblog.dynamic.application {

  public class Application {

    public static void Main(String[] args) {

      if (args.Length > 0 ) {

        Module module = ModuleManager.getInstance(args[0]);
        Console.WriteLine(module.getText());

      } else {

        Console.WriteLine("Usage: mono Application module_name");

      }
    }
  }
}

Now we can compile our ‘Application’ against the API…

application $ mcs -reference:../api/Module.dll Application.cs
application $ ls -la | grep exe
-rwxr-xr-x 1 michaelfclarke michaelfclarke 3072 Apr  5 19:32 Application.exe
application $

And now finally we can write a couple of modules!

using System;
using com.michaelclarkeblog.dynamic.api;

namespace com.michaelclarkeblog.dynamic.modules {
  public class Mike : Module {
    public String getText() {
      return "Hello from Mike's Module!";
    }
  }
}

using System;
using com.michaelclarkeblog.dynamic.api;

namespace com.michaelclarkeblog.dynamic.modules {
  public class Another : Module {
    public String getText() {
      return "Hello from Another Module!";
    }
  }
}

Compile the modules…

modules $ mcs -t:library -reference:../api/Module.dll Another.cs
modules $ mcs -t:library -reference:../api/Module.dll Mike.cs
modules $ ls -la | grep dll
-rwxr-xr-x 1 michaelfclarke michaelfclarke 3072 Apr  5 19:36 Another.dll
-rwxr-xr-x 1 michaelfclarke michaelfclarke 3072 Apr  5 19:36 Mike.dll
modules $

There is just one last thing we need to do before we can run our application. We need to copy the Module.dll API from the api directory into the application directory…

modules $ cd ../application
application $ cp ../api/Module.dll .

And now lets try running our dynamic application…

application $ mono Application.exe ../modules/Mike.dll
Hello from Mike's Module!

application $ mono Application.exe ../modules/Another.dll
Hello from Another Module!

application $ mono Application.exe
Usage: mono Application module_name

It’s been quite some time since my last post. This is down to a number of things - the first is that I’ve been really busy and so haven’t had that much time to write a blog post. The second is, with being so busy with mundane things, I’ve not really had anything worth while blogging about.

However, today I was on the phone with a friend (Anton) and our conversation slowly progressed to Dynamic Class Loading, specifically in Java. This made me remember that when I first heard about this technology (about three or four years ago) there wasn’t that much information available on the internet. So, I decided that I’d write a little (or perhaps not so little) post on how to go about getting Dynamically Loadable Classes in Java - it really is pretty easy.

The first thing you need to define is an interface for your modules. Mine is just going to be something simple that had a getText method to get some random text…

package com.michaelclarkeblog.dynamic.api;

public interface Module {
  public String getText();
}

You can now compile your API…

~/src $ javac com/michaelclarkeblog/dynamic/api/Module.java
~/src $

The next thing to do is to write your main application that is going to do the module loading. Mine is going to be a simple main method that accepts a java class name as a run-time argument.

package com.michaelclarkeblog.dynamic.application;

/* We must import the URLClassLoader and also our API. */
import java.net.URLClassLoader;
import com.michaelclarkeblog.dynamic.api.*;

public class Application {

  public static void main(String[] args) {

    Module module;

    if ( args.length > 0 ) {

      try {

        /* Create a new instance of the module.
         * Think of this as:
         *  Module module = new Module()
         * Except we're doing it at runtime 
         */
	String path = "com.michaelclarkeblog.dynamic.modules." + args[0];
        module = (Module)Class.forName(path).getConstructor().newInstance();

        /* Run the modules 'getText()' method defined in the interface. */
        System.out.println(module.getText());

      } catch (Exception e) {
        System.err.println(e);
      }

    } else {
      System.out.println("Usage: java application module_name\n");
    }

  }

}

That’s pretty painless, we can compile it now…

~/src $ javac com/michaelclarkeblog/dynamic/application/Application.java
~/src $

Now all that is left it to make our modules. I’m just going to write two, but you can write as many as you want.

package com.michaelclarkeblog.dynamic.modules;

import com.michaelclarkeblog.dynamic.api.*;

public class Mike implements Module {
  public String getText() {
    return "Hello from Mike's Module!\n";
  }
}

package com.michaelclarkeblog.dynamic.modules;

import com.michaelclarkeblog.dynamic.api.*;

public class Another implements Module {
  public String getText() {
    return "Hello from Another Module!\n";
  }
}

And now we compile our modules - notice that we’re NOT re-compiling the main application.

~/src $ javac com/michaelclarkeblog/dynamic/modules/*.java
~/src $

Time to run our program…

~/src $ java com.michaelclarkeblog.dynamic.application.Application
Usage: java application module_name

~/src $ java com.michaelclarkeblog.dynamic.application.Application Mike
Hello from Mike's Module!

~/src $ java com.michaelclarkeblog.dynamic.application.Application Another
Hello from Another  Module!

~/src $

Easy eh?!

I know the above doesn’t look very powerful. However, you can make some really dynamic and customisable applications with this technique where your users can even write their own modules for the application - all they need is the api/*.class files (or depending on how many you have you can jar them up)!

Tomorrow (simply because I am no tired) I’m going to post the same application and modules but written in C# and compiled using Mono under Linux. When I said earlier that it was hard to originally find this information for Java (which it appears not to be anymore) it was even harder to find this information for C#, let alone get it working properly!