Well, after running simulations with more nodes, I've determined that having all processes write to the same stdout terminal is hopeless. It's impossible to debug when you have more than a 1-2 processes interleaving debug printouts. Over the weekend, I started working on modifications to the simulator to have each node open its own console window when it starts. This removes the problem of interleaved print statements and makes things much easier to debug. Although it sounds easy, it is pretty difficult to open a new process in its own window. The only way I could figure out how to do it was to fork the process, send an exec command to open a new window, and have the node run as a standalone process inside the new terminal. If anyone has a better method, please let me know.

Anyways, the change means that I need to restructure the simulator and separate the node executable from the simulator executable. It's actually not too bad, and I've already done a lot of it. Since I was making modifications on the sim again, I decided to remove a lot of the hacks and actually try to turn it into a halfway decent application.

I was using pipes and the 'select' command to mux the data previously which was okay for a while, but started to get messy. Using pipes and select means that you're constantly polling the read pipe descriptors, and setting up the 'select' command is pretty ugly. If you're not familiar with the 'select' command, using it means that you have a lot of FD_SET and FD_ISSET macros in your function. In my opinion, the better way to do it is using threads to handle each of the pipe reads since you can just let them block instead of using the 'select' to poll them. It seems more natural to do it this way since the pipe communications were meant to block and the code flow is much better this way. I already converted one of the pipes to a thread and it made a big difference in how clean the code looks. So I decided to make the simulator a multi-threaded, multi-process application. Fun fun fun...

At least if I'm ever starving on the streets, I can try and get a job as a Linux system programmer.

Things are starting to look better now. After some elbow grease over the weekend and much research on Linux' interprocess communications, I was finally able to get my simple simulator up and running and connected to Contiki. The development process wasn't easy, but it took about a week (of hard work) to get it to the point where I can actually start simulating actual nodes. I'm actually quite satisfied with it. It's not as sophisticated as Cooja or Netsim (Contiki's native simulators) but it can do what I want and give me fine-grained control over how the simulation is run. Here are some of the details.

The simulator starts up as a command line shell. To add a node, you type "add". The simulator will then fork a process and run the FreakZ stack on it. The command shell also has two duplex communications channels to the node. One of the duplex communications channels is for data tx and rx. The other is for command tx and rx.

The node runs a simple command parser and is constantly listening to the command rx pipe for instructions. When the user types in a command and addresses that node, the command will be sent directly to that node. Once the node receives the instruction, the command parser will then parse the instruction and arguments and carry out the task.

Probably the toughest part was the radio medium. To simulate a radio medium in the simplest case, when one node transmits, all nodes should ideally be able to hear. I know, I know...it's impossible due to range, fading, etc...but I'm just talking about a simple case. When one node transmits, the data needs to be broadcast to all the nodes. The nodes will then check the address and determine if it should be discarded or sent up the stack. This is how a standard 802.15.4 radio works.

The broadcasting part was difficult because there's not really a good way to do one-to-many interprocess communication on Linux. Some of the candidates were "named pipes", shared memory, and client/server using sockets. However each of them had some drawbacks, mostly in complexity. I experimented a bit with some of them, but after awhile, I decided to take the easy way out. I just keep a list of all the nodes, and when one of them transmits data, it goes to the main shell which then forwards the data to each listening node. Its brute force and crude, but it was the easiest way to broadcast data to all my processes reliably. In the future, I'd like to add some features like transmission range or noise to the radio medium to make it more realistic. But right now, I just want to see how the nodes perform in a network setting.

So that brings me to today. I simulated my first node where I actually brought up all the layers and started a network formation request. Unfortunately, I found a bug immediately so I'm working on the fix now. Actually, it's strange because that same bug should have come up in my old test fixture (the single-process one), but I never saw it before. It's kind of a mystery that bears more investigation, but anyways it's better to catch bugs than let them stay dormant inside the code.

Damn. I just checked my watch and today is Friday. My last post, I mentioned that it was Friday night so I would crack open a bottle of wine and get drunk watching anime. Unfortunately, it was Thursday night. After leaving the job, the days just kind of blend into each other. My routine is basically wake up, miscellaneous, work on the project, miscellaneous, and go to sleep. I even lost my concept of weekends.

Don't get me wrong. I'm not a workaholic. I'm actually pretty lazy, but I'm just struggling to move the project forward right now. Apparently so much so that I've lost my concept of time. Now if I can work hard enough to go backwards in time ten years, it'd be great.

On the upside, life outside of the corporate world is excellent. Aside from the lack of insurance and the worrying about how long my savings can hold out, I must say that I've never been happier. I'm writing this entry on the veranda of a cafe overlooking a park right now. Its noon, the weather's beautiful (a rarity in Tokyo), and there's a lunchtime orchestra concert going on over there, injecting a little culture into this ghetto engineer's life. Also, I can see a bunch of suits watching the orchestra but getting ready to leave since their lunch break is over. Ha ha ha. Oh, the price of an eternal lunch break. 

And the best part is, my crappy little hacked simulator is showing some signs of life. Ah, the little things that keep engineers and programmers happy...

Have a nice weekend (this time it's for real)! 

I was actually able to pull it off. In  my last devjournal, I expressed some of my uncertainty in being able to port the FreakZ stack over to Cooja. Well it took a lot less time than I expected to get it working under the simulator. There are still a lot of things that need to be done, however I'm almost 100% confident that I can get everything up and running now.

Don't get me wrong. It was anything but easy. It was like a warzone in my development directory after I was finished. Porting a stack under Cooja requires hacking C, makefiles, and Java. Actually the makefile hacking took the longest because there were a lot of dependencies that I needed to take care of as well. But in the end, I was able to get the stack to compile and be recognized by Cooja.

There were also a lot of times that I was completely stumped, and a few times where I just wanted to cry. I ran into some issues where I would have name clashes with Contiki (ie: mac.h exists in Rime and in FreakZ) and assert.h is overridden by the Contiki version. I also had a problem where everything was correct except for a space in my makefile. That one almost killed me. Anyhow, things finally worked out in the end and Cooja can now signal events to the FreakZ stack via Contiki.

The things left to do now are to modify the interfaces and plugins. I need to start examining the radio code and the radio monitor to see how much changes will be required so that I can start transmitting and receiving to other nodes. I also need to figure out how I'm going to start up the network, since I can't just have all nodes turn on at the same time. But at least now I have more confidence in my ability to get things working with Cooja which is a huge plus.

Here are some screenshots of the FreakZ stack working under Cooja.

I'm currently doing battle on two fronts. On one, an onslaught of cold viruses seems to be attacking my immune system and rendering my body immobile. Luckily, there is only a slight haze in my head so I still have the ability to ponder the second battle. That is, my battle with Cooja. I've been spending the past week studying Java and poking through the Cooja source code. I was pretty intimidated at first because Java is not a strong language for me at all. It's been a long time since I had to do object oriented programming, and that was back in the college days when I spent most of my time ditching classes and going to dance practice. Cooja is written almost completely in Java and interacts with Contiki via the Java Native Interface (JNI). The JNI is like a translator that allows Java to call C functions and vice versa. 

So here I am again with my Java books, reading through the source code of the Cooja simulator and trying to figure out how it works. There is a user manual available for the simulator, but it's a little bit thin. I think at a bare minimum, it takes the user guide, the PPT slides , and Fred Osterlind's (the author of Cooja) master thesis paper on the simulator to get an understanding of how it works. I've already been through those documents a few times.

I feel pretty good today. It feels like I was able to accomplish a lot of things on the stack for a change. Sometimes I feel pretty lethargic when I code. It's like working out; hard to start, but once you get going,  it's easy to lose yourself in it.

One of my major accomplishments was rewriting a lot of functions to remove some useless code. In the 802.15.4 and Zigbee specs, they use a large configuration structure called the Information Base (MAC Information Base (MIB) and Network Information Base (NIB)). They also specify how to access it, which is using set and get functions. The problem is that the structures are quite large and contain varied data types so the code to set and get all of the struct members is quite big.

I was thinking today if it was really worth the code space to have set/get functions instead of just accessing the structures via a pointer. That was when I decided to check the code size of the object files. The total was 3k of flash for the set/get functions. Holy Scheiser!

I sharpened my axe and started slashing left and right. Basically, I removed all of the code in the set/get's and even deleted the files. Then I just wrote a simple one-line function that would return a pointer to each of the structures. I had to do a lot code surgery in other areas since I had the set/gets sprinkled all over the place. But in the end, in return for accessing the structure via a pointer, I got an extra 3kB of code savings. Don't laugh...that's 10% of a 32 kB flash MCU! Wow...I guess I am a geek. Anyways,  accessing the structures directly is a bit unsafe if you're not careful, but I don't think set/get's add much more safety either.

In other unexciting coding news,  I put the finishing touches on the network discovery and mac scan functions and wrote a self-checking test to make sure it worked. After debugging a couple of bone-headed mistakes, I was able to get it to pass the simple test I wrote, as well as the regression where I run all the previous tests. That felt pretty good. 

So off I go on to the network join functions. This one will take a while because there is a client side and server side to the join procedure and both of them are pretty involved. I'm also gonna be a bit busy this week since the part-time job is going to take up at least two days due to unforeseen busy-ness, and I'm spending one day to go to the Sensor show at the convention center in Tokyo. Hopefully, I can take some cool pictures and show everyone some of the new sensors that are coming out over here. 

That's about it for today. TTFN.