Andrew's workblog and experiences in software development.

28th July 2014

Post

The Muftec Fabricator

Something I’d been planning on writing since the start of the project, the Fabricator is a surprisingly small (~222 sloc) class that takes the lexer output and converts a MUF script into an executable MSIL application. (It’s actually a compiler, but it’s called the Fabricator because the lexer is called the Compiler and I really need to fix their names)

In order to achieve this, where the normal runtime executes code by lexing the source, putting tokens on either the execution queue or the runtime stack, then starting to run commands in the execution queue, instead the Fabricator translates the execution queue directly into MSIL commands. Commands and stack inserts are still run in the same order, but functions are called directly instead of performing the lookup at runtime, resulting in what should be the most efficient code without Muftec stack items to being native CLR types on the stack.

In effect, a MUF program turns from

: main
    2 2 + print
;

into (C# for readibility)

public static void func_main(RuntimeStack stack) {
    stack.Push(new MuftecStackItem(2)); // 2
    stack.Push(new MuftecStackItem(2)); // 2 2
    BCL.Math.Add(stack); // 4
    BCL.IO.Print(stack); // Prints '4'
}

Now the application can be optimized (as much is possible with this exeuction mode).

Optimally, we’d use native CLR types like int and string instead of wrapping them in a MuftecStackItem, but doing so would mean we need to dynamically generate MSIL functions instead of doing the normal execution stack flow. I think this is certainly an alternative, but I wanted to get a MSIL compiler up and running in the simplest form possible without sacrificing the rest of the project, especially since the execution mode is a lot easier to debug.

There are a couple components needed to write new applications in MSIL. The first is provided by the .NET framework reflection tools as the ability to create a new Assembly, Class, and Methods through code. Once you’ve created a method, you can Emit IL into the method. Normally you would accomplish this with the Emit class which involves calling functions to add MSIL bytecode manually.

Instead of using the baseline IL Emit, I chose to use a project I came across called Sigil. This allows me to write IL instructions more cleanly, for example:

In Sigil, calling 2 + 2 and printing would be:

var funcDef = Emit<Action>.NewDynamicMethod("Add");
funcDef.LoadConstant(2); // Put the int 2 on the stack
funcDef.LoadConstant(2);
funcDef.Add(); // Adds two numbers together
funcDef.Call(typeof(Console).GetMethod("WriteLine", new[] { typeof(object) }); // Get the Console.WriteLine method and call it
funcDef.Return(); // All methods must return
var del = funcDef.CreateDelegate(); // Compile the IL and turn this into a real usable delegate
del(); // Execute the method we just created

where as doing this with IL Emit would be:

var funcDef = new DynamicMethod("Add", /* some other stuff */);
var il = funcDef.GetILGenerator();
il.Emit(OpCodes.Ldc_i4, 2); // Put the int 2 on the stack
il.Emit(OpCodes.Ldc_i4, 2);
il.Emit(OpCodes.Add);
il.Emit(OpCodes.Call, typeof(Console).GetMethod("WriteLine", new[] { typeof(object) })); // Get the Console.WriteLine method and call it
var del = (Action) funcDef.CreateDelegate(typeof(Action));

It’s very similar, but requires knowledge of specific opcodes, whereas Sigil provides several conveniences especially including compile type sanity checking (like knowing a function takes 2 arguments but only 1 is on the stack) which helps a lot when learning how to use it.

An interesting way to look at how this works is the fact that the GenerateInnerFunction method of the Fabricator is actually doing exactly what the Run method of the MuftecLibSystem does. That class actually performs execution, whereas this class pretends it’s executing the application but writes IL to do it instead. In the Fabricator, however, instead of jumping into a new stack when we move to a user defined function, we’re actually writing a real .NET function that we call, so instead of being recursive, we just execute that defined method instead, which simplifies execution flow a bit as well. This also required me to create function definitions before defining their contents, sort of like declaring headers in C, since they can’t be called if they don’t exist.

I haven’t yet done performance testing, but despite still needing a runtime stack, the lack of an execution stack, not needing to resolve function or opcodes by name at runtime, and other optimizations should make this method a lot faster. There are still a few ways to improve performance even further, especially if I make the whole system run off dynamically generated methods instead.

I have a few things left I need to do with Muftec at this point. Primarily I need to finish writing all of the opcodes, as I want as much parity with Fuzzball (the source of our implementation) as possible, and also need to add some other basic features like loops. If-else statements are supported, but not much else.

Tagged: mufteccodingcsharpcompilers

6th January 2014

Post

Progress Slowed

I’ve been busy with a lot of annoyances like real life, but I’ve completed a rewrite of the database generation script for MyConbook, which introduces multi-convention support in preperation for the iOS app, as well as a redesign of the Android app I’ve been working on in my spare time. I’ve also been working on the side with an art site (which will remain unnamed) as a part-time developer, which has been eating at some of my extra outside-work development time.

I have a couple blog posts I really want to get to, including some about project planning and organization (similar to my Trello post, this time about OneNote). I’ll have to get to writing those soon, but I’m not dead.

Tagged: arguably_lazy

11th November 2013

Question

Anonymous said: Hello Andrew, I start developing an android application and I arrive to check the if the airplane mode is changed and this work fine . but now what I want to know is the event on checking the checkbox of airplane mode . I get the event of airplane mode change but I want to do something when user click on airplane mode and before connection change to airplane in other way I want the event on keypress of airplane mode checkbox. I 'm waiting with please for your response.

Sorry, this isn’t possible! I assume you’re already listening to the broadcast event for the airplane mode change, but that’s the only thing an application is allowed to do when someone is switching into airplane mode from the settings menu.

8th August 2013

Question

Anonymous said: yes i was wondering how to make a bot like your groompbot but make it post things from a different channel and reddit

Groompbot is open source, so you can already use it and configure it for a different YouTube channel and subreddit.

If you go to https://github.com/andrewneo/groompbot you can check out the project, and see what you need to get going. It’s as simple as installing gdata and praw for Python, setting up the configuration file, and running the script at a regular interval.

4th March 2013

Video

I’ve been on vacation for the past few days and have put a lot of work into MyConbook for iOS. I’ve uploaded a video of my progress, have a few pieces left to do (including graphics) but at this rate I shouldn’t be long before I can submit it.

Tagged: myconbookiOS

3rd November 2012

Post

Trello

I’ve started using Trello for task management. Before, I would just have a huge list of things I wanted to do seperated by task in Evernote. This is okay to an extent, for simple lists, but I’ve found Trello is a wonderful way of going about managing tasks.

I’ve moved a lot of these sorts of lists into Trello, including for all my software projects as well as several other personal things (like stuff around the house, day job items, etc.) and find it to be a very well structured way of organizing my data. I like how instead of just having a basic “todo” list, I can customize each board however I like, allowing multiple categorizations, different approperations of tags (the colors), and especially multi-user support. I could so see a team working with it as a “to do” list instead of purely a bug tracker. Each item can have comments, due dates, images, even multiple checklists. The concept of moving a todo list out horizontally makes it a very powerful way of organizing data.

The screenshot above is for MyConbook. The layout for that is a pretty standard todo list, but if I wanted to I could create separate lists for upcoming conventions, Android, iOS, Web, etc. and organize everything independently. Not restricting me to how the application feels I should do things makes me want to do everything this way. I had been considering setting up a bug tracker just to track these sorts of things. I feel that this lets me do both - each project can be both a to-do list and track bugs I want to take care of. If a project has an external bug tracker, like on say Github, I can still add an item here and handle public communication and resolution externally.

Tagged: metatask management

2nd November 2012

Post

The Upgrade to Tablets

It’s been a long while, unfortunately with no progress on the iOS version of MyConbook. But, after Google released their Tablet App Quality Checklist I figured it was time to do what I had thought about starting around June, and implement a tablet UI for MyConbook for Android.

When Android released Honeycomb (version 3.0), they added a UI concept called Fragments. Fragments are these neat, versatile elements in Android because they can be used in a number of ways - from simply being an entire Activity, to being a portion of an Activity, to just being a code block with a lifecycle. Fragments are simply classes that can be added to a layout (a view is actually optional), so this way you can have independent UI pieces in play at once.

I’ll get into Fragments themselves more at another time. For MyConbook specifically, a bit of a task had to be undertaken. I figured the best way to display the UI for tablets would be to have two split (~30%/70%) panes when a tablet device is landscape. This would usually involve a list on the left side, and a details view on the right, though this wouldn’t always be the case.

So to start, this meant I had to convert all of my existing Activities to Fragments. This isn’t too hard, but there are special things you have to pay attention to in the activity lifecycle that are different for fragments. You can get away with leaving them alone if it’s the only thing being called, but when you start to get into reuse you have to make sure your UI is being built in onCreateView instead of onCreate, for example. Converting most of the elements wasn’t too difficult, especially as many of them were already at least FragmentActivities, since MyConbook uses ActionBarSherlock.

Turning existing UI elements into fragments is one thing, displaying them at the same time is another. Normally, from one activity, you would simply start an Intent to display another activity, and pass along the arguments to be displayed. However, if you’re showing more than one fragment on the screen, the changing fragment needs to be created and informed via the parent activity. So, you would enter an “activity” like the schedule, and it would initiate the day select list in the left side fragment. Tapping one of these elements would normally create another activity, but in this case we want to show another fragment on the right, so the fragment code is created and the message for which day was chosen is passed along by a function, instead.

Creating and message passing by a parent gives us an advantage. Due to the way Android handles view layouts, if we’re not on a large screen in landscape view, the right side fragment container will not be inflated. So we can tell at runtime that it is not there, and instead replace the existing “left side” container (the only one, on any other size/orientation) with the new fragment. Fragments have automatic back-stacking, so we can pop it off and go backwards automatically without having to manage it.

I’ve implemented this for a few of the activities, and it seems to work rather well on both phones and tablets. Overall, however, I’d like to move to having one activity where the “action” is simply determed by a dropdown in the ActionBar instead, and never change layout away from that default split mode unless rotating the screen. This would be a little more complicated (I haven’t gotten menus in the Action Bar working yet, as is, for example) but with properly designed code it should be easy for this single activity to initialize an “action” and allow each action to handle its own tasks as is.

Right now I don’t have much in the way of pictures to help this make more sense, but once I’m closer to being done I’ll try and add some code snippets to make sense of it. Moving to Fragments and a split UI has been rather intimidating to me, since it requires a lot more code than plain activities do, but after a point it just starts to click.

Tagged: myconbookui

1st November 2012

Post

Muftec

I meant to write this up a bit ago, but I wanted to write something short (since it’s still in progress) about an older project I’ve gotten back to, called Muftec.

At it’s core, it’s a Forth-ish stack based programming language, shamelessly copied from MUF (Multi-User Forth), specifically from Fuzzball 6. However, this specific implementation intentionally does not have any language constructs from the MUCK itself, like dbrefs.
I decided to undertake this project since I had never taken a compiler course, and I figured Forth is a rather easy place to start, given its stack-based nature. I was already familiar with writing MUF (though it’s been a long while) so I thought I would throw something together.
Muftec is written in C#, and relies on reflection during the compilation stage to register opcodes (external functions). Right now, the application is capable of taking a file (or just a string) and compiling it into an “execution stack”. It is parsed to tokens at compile time, and then at runtime individual values (like strings or integers) are placed on the “runtime stack”, to be used by opcodes, or if it detects an opcode, the associated C# function is called out of a registered class library. The system is designed to be able to be expanded so an implementing application could take user Forth and the application’s opcodes would be callable.
At the time of this post, a majority of the important features have been implemented. Most of the relevent MUF opcodes are there, functions work, and if statements work. Still missing are library includes (and other preprocessor abilities), arrays and dictionaries, and their related opcodes. Global variables are implemented but not tested, and local variables and variable scope have not yet been implemented.
The compiler is built as a state machine (inheriting from a common interface). There is a main "compiling" state, which knows how to read the outer Forth syntax. When it detects a Forth function, the state machine begins to read the inner function code, which processes elements such as variables, if statements, opcodes, integers/strings/etc., and the like. An if statement invokes another instance of the inner function code reader, which then creates a special execution stack element so the if statement can be evaluated at runtime by C# code. It also registers Forth functions (as a dictionary) and declared variables, so the runtime can keep track of them to be called. Functions and the execution stack are built as Queues, which are popped as they are executed. From what I can tell of traditional execution environments, they should probably lists with a program counter, but I don’t feel the need to implement them that way unless I need to be able to read past execution code.
Running the compiled code is rather simple (see the Run function in this file), it takes the execution queue (the compiled code), a runtime stack (usually empty at start) and the registered callable functions and variables. Then each element on the execution queue is evaluated, which either means adding a static value to the runtime stack, or doing something like calling an opcode. Opcodes are registered at the start of the environment, and found at that time by reflecting for a given OpCodeAttribute attached to a function. Opcodes are added by name to a dictionary, with a delegate assigned to call when hit in the execution queue.
Individual opcodes are rather simple. They receive an instance of the OpcodeData class, which contains persistant data that can be passed through between opcodes. Right now this only really only contains the Random seeds. This object however also contains the current runtime stack. Let’s take the simplest example as a demonstration on both the execution environment, and how the code handles it:
The Muftec code:
1 2 + print
During execution, this first adds 1 to the stack, then 2. Then the opcode + is looked up in the lookup dictionary, which calls the Add function currently contained in the Base Class Library provided by the project. The Add function pops the top item off the stack (2), then the second (1), then adds them together. It takes this new value (3) and puts it back on the stack. In our example, the print opcode is then called, which would read the 3 off the stack and print it to the console.
Functions are defined as:
: functionname
  <content>
 ;
There must be at least one function in a script, as the last defined function is called automatically. Functions are given names, and because function registration happens before/during code parsing, a function must be defined before it can be used, much like in C. This can be avoided if we just check for a function name at runtime before assuming an unknown name is an opcode instead.
And that’s all there is to it. This writeup only covers the absolute basics behind the concepts of the language. Because it is intended to mirror MUF more directly I don’t really plan on writing more about the semantics of the language, but I may write more in the future about challenges I have or will run into working on more parts of the implementation.

Tagged: muftec

8th September 2012

Photo

Okay, one last post. Just wanted to show a diagram of the storyboard for the iOS version. The Storyboard feature is really neat. I hated doing UI work before they added this, it just seemed convoluted. I&#8217;m sure it&#8217;ll get worse when I actually get into talking to the UI from the code. Designing the workflow is a lot nicer now, though, at least, and I can see that it works even without data by running it in the Simulator.

Okay, one last post. Just wanted to show a diagram of the storyboard for the iOS version. The Storyboard feature is really neat. I hated doing UI work before they added this, it just seemed convoluted. I’m sure it’ll get worse when I actually get into talking to the UI from the code. Designing the workflow is a lot nicer now, though, at least, and I can see that it works even without data by running it in the Simulator.

Tagged: iOSMyConbookUI

8th September 2012

Post

Git that Git

In addendum to the previous post, I was reminded that I wanted to write a quick bit on using Git. I mostly wanted to mention, that instead of dropping the source for FMDB into my project, I included it as a submodule so the code is pulled in from the Github repository when you clone the project. This can help with licensing and stuff when I get around to that part. I’ll talk about licensing later, though I do note I have plans to make the source available for the Android, iOS and web versions of MyConbook.

For Git in general, I find it rather awesome. I used to use SVN a lot, and was confused why you would want a distributed source control system. There are plenty of blog posts that will explain Git better than me, but I find it much more valuable, particularly just pointing out the fact that your local repository doesn’t affect any of the others - so you can go ahead and break yours, create branches, merge like crazy, and only send back what you mean to send back.

Both Eclipse and XCode have a good integration for using Git, and no matter what source control you’re using, having it is better than nothing. I’m a little weird and don’t like to commit a project until I have something that “works”, even if it compiles. Otherwise I feel that I’m just going to be committing a lot of changes as I work out how I want to structure an application, and I could have saved time and trouble by waiting. I can miss out on some things like wanting to see how I did something in the past, but the turnover is so short at this early stage in a project that it’s not really worth it. That’s just me, though.

Tagged: metaGit