Happy Holidays and Current Progress

Here's a late Happy Holidays and a Happy New Year to all who read this.

This will be my final update of the year. Like I said in my last post I started from scratch, but I did quickly run into some problems. Everything is so interconnected and interwoven that going the route I was just isn't make sense any more. So I'm back to my prior method of taking things apart. Once, I've got sound and graphics completely separated out then I'll work on separating out the RSP and the Dynarec/CPU last of all. Then I can take the code I cobbled together and start bring the other parts back in one at a time.

I've been rather busy this month with the Holidays and my schedule overall won't be letting up anytime soon, but I will be working on the code in what little spare time I get. My next post will likely be around the end of next month. Hopefully, I'll have more news by then.

Rev 444 and the beginnings of Beta 3

Rev 444 was just regression fixes and a bit of code clean-up from some of my prior commits. Now, I'm on to the major recode for Beta 3. I've changed the way I'm going about it from my prior post here. I've decided instead of working in reverse removing parts of DX64 and working backwards, I will start from scratch, import code as needed, and work forward. This way I can ensure all the code gets looked at and I would've basically been doing this after I finished with the code disassembly.

I've talked with Strmnnrmn about how I plan to do the ME move to get a normally serialized process working asynchronously. Right now in DX64 it goes:
1.Dynarec
2.CPU Execution
3.Graphics
4.Audio (Although in Async this is basically bypassed on the SC.)

Which is basically serialized access to the individual components. To make this a truly multi-processing app. When the move is done it will go:

First Two Frames:
SC                      | ME
1. Dynarec (Frame 1)    |
3. Dynarec (Frame 2)    | 2. CPU Execution (Frame 1)
4.2 Graphics (Frame 1)  | 4.1 CPU Execution (Frame 2)
4.3 Audio (Frame 1)     |

Subsequently: (Frame X = Current Frame, X-1 = Last frame)
SC                      | ME
1. Dynarec (Frame X)    | 0. (Finishing execution of X-1)
2.2 Graphics (Frame X-1)| 2.1 CPU Execution (Frame X)
2.3 Audio (Frame X-1)   |

This will mean at any given time DX64 will be working on two frames at once. I want the Dynarec, CPU Execution and Audio to all individually be able to be moved between the SC and the ME through build flags as needed for debugging purposes. Graphics will always be on the SC as the ME has no GE access. (The only exception there might be some math heavy tasks like clipping, but without a VFPU I think this should stay where it's at.)

What all this means is I will basically be recoding everything from scratch and looking at all of the code for optimization and memory usage as I go. This includes uCodes, graphics, audio, the combiner, RSP emulation, etc... I literally mean every line of code. I got the go ahead from Strmnnrmn to do this, but I'm not going to be committing the changes as I go as not to break the SVN. I will also be looking at the new GUI and preparing for it as well as all the other plans I had listed for Beta 3. Next time I post, I will update my progress on all of this and list any issues I'm running in to.

Rev 443 and the next few revs

For Rev 443, I tracked down the issue plaguing Paper Mario. Since it was a regression, I went about it the way I normally find regressions I found the last revision that didn't have the regression and I checked the changes to the next revision after looking at the purpose of that particular revision. In this case it was rev 431 and 432 respectively. However, Rev 432 had 50 files and a couple hundred lines of code changed. So, over the course of two days I went through each file one by one and it turned out to be one line one 1 file which was the last one I went through. In Rev 432, I reverted some changes from some older Revs to fix a few regressions causing lock-ups in some titles. Now, I need to finish going back through the changes and see which are needed for speed and preparation for the ME move. After that, I have a few quick ideas I'll look at for further speed. After they are implemented I'll explain them here. Hopefully after the speed regressions and my new ideas we should see an overall boost in speed. Then it's on to the Async bug and the rest of the changes leading to Beta 3.

Rev 435 and upcoming...

Well lately I've been trying to track down why when a specific "bug" happens we get about a 46% increase in speed on Mario 64. In the process of trying to track it down. I noticed a few things that either got reverted in 432 and shouldn't have and 1 more small speed-up I noticed where logic on a function could be improved. The speed-up seems to be fully dynarec related, but will take more time to track down. Next I plan to see what part of the 432 changes caused Paper Mario to start freezing so much. Stability (or consistency ) of games the play is definitely needed before we can proceed to move the CPU to the ME. I added back the reverted code and another speed up or two for 435. I also added an advanced options menu as a secondary menu to the ROM options. These are all the extra items that can be enabled in the roms.ini file. Most of them cause breaks in many titles which is why they are on the advanced options, but you can try them out to see if they help the speed or graphics in other titles besides the ones they are specifically written for. Enjoy


P.S. Sorry this post is late, but the Rev has been posted for a few days.

Rev 432

I just posted Rev 432 here's what happened. I had a few speed ideas, but thought ehhh... I'll throw in a fix for GoldenEye(GE) freezing since it's a regression (I.E. It didn't used to freeze.) just for the hell of it. Along the way of doing that (I found where it broke, but just reverting that one part broke other games) I remembered having to disable the loop optimization (A speed-up Strmnnrmn added for R14 that had some compatibility issues.) So, I decided to check what broke it and found that as well and I found it. All in all code from 3 revisions (All that Howard0su posted way back, 305, 316, 318) needed to be reverted, some of it I was still able use use and the rest will be brought back over time as we near Beta 3 and I track down each piece that was causing the breaks.

In the end, 432 includes: GE(No Freeze) + TS(No Freeze or Crash even on Phat) + Loop Fixed + Minor Texture Speed-ups + A few new settings for texture update. (It now has: 1, 2, 3, 4, 5, 10, 15, 20, 30)

The extra speed is enough to be noticeable(About 5%): The Mario head on Mario 64 goes at a consistent 19 FPS. Games the dynamic loop never worked on will still get some extra speed, but not as much. (It still freezes MM and a few others.)

For the next bit of code I post I plan on working on how DX64 handles Textures, on the Phat and even sometimes on the Slim DX64 runs out of memory for texels in both the VRAM and the secondary RAM buffer. This was my purpose in reducing the secondary buffer on the Slim a few revs back so I can do testing with the lower Phat value. Besides trying to get a bit of memory from else where (Some other buffers are a bit larger than needed.) I plan to work on the way Daedalus decides to throw away a texture when it's no longer needed. I thinking that I will have it throw away large textures sooner. Some sort of texture compression may be another possibility.  A couple games that are good examples of this are Transformers - Beast Wars, Tetrisphere, and Space-station Silicon Valley. All three have issues with texture memory with Beast Wars being the worst. This also causes the crash where if you go to the select menu several times the emulator crashes. I reduced the memory intrafont uses a few revs back to help with that. Right now DaedalusX64 keeps textures for 10 Frames automatically (Not 100% sure if this is 10 actual frames or 10 virtual frames). If I lower this to 1 Texture update is not needed as textures are all thrown away every frame, but then there is a speed cost. However at even a setting of 2 Texture update is need for persistent items like onscreen text it OoT but there is still a speed cost. Loading small textures take a small amount of time compared to large ones, but loading them too frequently causes cache thrashing. So in the end we need a more dynamic way of dealing with this. Also as an added measure large textures will likely get converted to 16Bit to cut there size in the memory in half which would help as well. This fix in the end should help a few crashes and possibly add just a bit of speed as more textures (Especially the small ones) will be stored in the VRAM (which is 2X Faster than the RAM) Also as part of this I will be testing various games to get a count of how much texel memory they are using at any given time to know how much I need to save. The fix for the Slim it's as simple as upping the secondary RAM buffer, but as you can see it's going to be a bit more complex for the Phat.

BTW: On a side note my crazy plan from a few posts back didn't happen... It will eventually just not yet. Also my wife's desktop died so until we get her a new system (which may take a while.) we will be sharing my laptop on which I do my developing. Other than that I will continue to try to udpate this around once a week to once a month depending on progress I make and life happenings.

A week away and forums down

I've been gone for about a week due to an elderly family member having heart surgery (It went great). I just got back today.

I noticed Dark-Alex took his forum down, but he has stated M33 isn't dead and I quote from the IRC "Forums closed down due to expenses. M33 is still alive and well. Don't believe or spread rumours! | forums.exophase.com"

Our situation isn't due to money but donations are still welcome, but it is due to a problem with our host. I have decided we are going to switch (Don't know about costs yet). I only have yet to decided to whom. I am looking at this now. I will likely purchase the new hosting myself and a friend wants me to set up a website for his business. (Which may offset the cost, but all new donations will go to hosting as well as the ad revenue. Anything beyond that will go to charity.)

Side Note about current code efforts: Soon myself, Strmnnrmn, Howard0su will all be working together on the ME move issue. Howard0su and Strmnnrmn have been busy with other things and will be available to help soon. Howard0su has given me his latest WIP code along with Strmnnrmn. Myself and Salvy are working on the memleak. I'm working on some of the crashes. Grazz is still mostly concentrating uCodes for compatibility. So, compatibility, speed, and crashes are all getting looked at.

Rev 424

Well I've just commited and posted Rev 424. Here's the changelog:
[+]Renenable SIMULATE_DOUBLE w/Mario Party hack
[+]Reduce resource usage of Double Display w/o breaking roms
[+]Removed need for extra sceGuSync for FPS Display
[+]Switched from string compare to flags for hacks
[+]Added hack flags to Roms.ini and as menu options Debug mode
[~]Added files the Vis C++ project to reflect Changes in Microcode Structure
[!]Fixed 007 Broken on Rev 423
[+]Reduced extra rendering steps on non-rendered frames with FS enabled
Note: Approx 10% Speed-up, some games that need DoubleDisplay to function correctly have issues under FS (Ex:Automobili only work properly under odd FS)

A total of 16 files were changed. As promised most of the speed we lost between 300 and 423 should be back now and hopefully without side effects. All you GoldenEye fans should note the GoldenEye sky uCode has been disabled until after the release of Napalm's source code or I find a way to fix it myself, but at least it's playable again after Salvy broke it. The speed increases will be more noticable in some games vs others. Next I'll be back to working on the CPU to ME move. Also don't forget to overwrite the roms.ini when you upgrade as new stuff has been added to it for per game hacks.

Back to coding and the recent downtime

As many are aware and is still a bit obvious, the forum went down over the weekend. This was due to an admin who had some personal issues and a forum issue that pushed him over the edge. Luckily, the databases were still intact and functionality has been restored. Measures are being taken to prevent this in the future. Hopefully the theme should get restored today and we should be back at 100% then. I will leave it up to the guilty party/parties to provide any further details.

Life has been busy for me lately. The crazy thing I had planned didn't work out so there are no pictures to show off, but there are plans for it down the road. (It has nothing to do with DX64 though.)

Things have settled down in my life a bit and I can get back to coding. I'll be doing more code clean-up and finishing up restoring some of the lost speed. I've just got a couple of double display issues to work out and the next Rev I submit should have a 10%+ gain in speed with at least current compatibility. After that it will be back to ME testing unless the GoldenEye sky fix in Napalm gets released first.

A recent loss and my lack of updates this week...

I've been away this week due to friends who are practically family being in town after a tragic death in their family. He was really a good kid. I can still remember playing N64 with him when he was 7 at my wedding there in Denver. His mom is my wife's long time best friend and she helped raise him as a baby. So he was basically her nephew. Due to this I won't be working on DX64 until next week.

I have a couple speed fixes I haven't committed and should be back. I'll be fixing all the speed regressions I've found between Rev 300 and Current. Also be taking care of a couple of minor things as well. Napalm, the Glide plug-in that has the proper (Rice's isn't a "proper" fix and more of a hack) fix for GoldenEye's Sky should be releasing it's source code soon. When it does I'll be using the from that part to give us the fix. There is also more we can still get from Rice. I also have other code I need to fix and get working properly and run Media Engine related tests to make sure we use the fastest most stable method for sending and receiving data between both of the PSP's processors. Then I can finish my ME work.

Lost and Found...

I was comparing the FPS in various revisions a while back and noticed several revs had a drop in FPS from prior revisions. So, today I'm building an FPS database from past revisions going back to R12, R13, R14, Beta 1, Beta 2, Beta 2 update, and also the alphas (about every 25 revs). I'm going to be testing 2 different areas in about 5 games per test and recording the data. Now this amounts to about 250 tests total, but I believe it should in the end net us at least 5% in FPS if the numbers I saw in prior testing are correct.

After this first round of tests and a speed graph of sorts is put together. I will look for where for any losses and try to correct them. We have a lot of code and around 500 or so changes that have been made since R14 so I think this is the best approach. After correction and in the future. I will run these tests on every new revision. This will take me a few days just to finish all the testing and I'll report the results then. Then report again after I correct as many losses as possible.

Cleaning-up

Well a couple of days ago together myself and Grazz have cleaned up almost all of the release mode warnings. We went from 90+ to 6. Next I'm going to work on the warnings in various compiler/debug options. Once I'm done with that, I'm going to help Salvy some on the uCode side of things a bit. Then it's back to my rebuild. I've also got more code from Howard0su to look at, but I'm going to be busy over the next few days without much time for DaedalusX64 which is why I'm only going to be working on small things like clean-up. Next week I should be back in to full swing working on the rebuild and will post another blog entry then. Also I plan to post a few things not DX64 related in about 2 weeks that will show you a little about me and just how crazy I am.

Headed in the right direction...

Well, I had a voice chat with one of our currently inactive devs who got the ME idea going first(Howard0su). He liked my plan of attack and my general idea to get it going. I also found out using the method he used, he did meet with limited success. I think by going about it using my method, it should end up more stable and work better in the long run which is what I'm aiming for. Tomorrow after I take care of a few things, I'm going to start the disassembly part of my work. I'm hoping this will also give me a better understanding of the code overall as well. I also heard about some new work being done with the ME concerning signaling which hopefully we can benefit from in order to have less overhead.

Day 2

Well as far as I can tell I've removed all non-PSP related code. Including endian and W32 related stuff. Next is the disassembly of the individual parts. First the Sound, then Graphics, finally Controls/SaveTypes. Then I'm going to debug a few nagging menu issues(Namely the alphabetization issues). Afterward, I'll begin the process of writing the new thread management code for Daedalus. Then after that is done and I have the ability to run the CPU on either the ME or SP, I will bring back the Controls/SaveTypes first(And work to get as many games as possible booting while the CPU is on the ME or SP). Next will be graphics then I can begin running performance numbers(after writing my new profiling code to better work with the new system). I hope to be at this point in a few weeks to a month, but it might be longer. I have a few things coming up next month and one in particular that will take me away from DX64 for a week. On top of that as I bring each section back in I'll be looking at the bugs in each and doing what I can to clean up, comment, and optimize the code, so in the end this part may take a while, but I'll keep you informed here.

Getting Started

Today I decied to get started on the tear down and rebuild of this project. The first thing I did was remove all traces of XBOX and OSX support. Next will be Big Endian support the DAEDALUS_xxx platform and then I will back up the project at that point before I removed the W32 support. Why? Just to make it easier to reimplement it once I'm done with all my work recoding specifically for the PSP. I still want to support windows for debugging, but not until I'm finished on the PSP side. In this next stage all Defines will be looked at. When I'm done the code will be pure PSP homebrew. Then I will start taking apart and separating out each piece in order to rebuild it from the ground up. It's easier to get it working with one part at time then to try to do it all at once. It is a slow but very methodical route I'm taking so that each piece gets looked at worked on and improved in the process. I personally know nothing about programming for the XBOX, OSX, or Strmnnrmn's unnamed system, but W32 and the PSP I can handle. I'm only working with the PSP part first as this will be the primary platform with the others as more of an afterthought to give devs/users more options to work with down the road. Will this possibly be put on platforms other than W32 and the PSP in the future... maybe but I'm not the one that's going to do it. Well it's back to the code for me now.

Restarting DaedalusX64

That's right I'm restarting DaedalusX64 practically from scratch. I've already announced it to the devs and my reasoning behind it.

In preparation to move the CPU(R4300i) to the ME(Media Engine) I noticed we lack encapsulation of CPU. Howard was trying to accomplish it but was unable to finish it. So, due to this I'm going to be semi-restarting DaedalusX64 from scratch in order to encapsulate each component to be handled by a central thread management function. It would be like a PC emulator having separate (CPU w/Dynarec), GPU, SPU, and Input plug-ins and the "main app" only handling the I/O between them instead of the CPU built in to the emulator. In our case the "main app"(AKA the thread handler), the GPU, and input will run on the SP, and the CPU on the ME. The SPU will be able to run asynchronously on either. (So it's portable to where we need it.) By having each item fully encapsulated we should be able to debug CPU/Async Sound issues by bringing it back to the SP with a simple #define flag since they will be fully encapsulated and not dependent on other functions to operate. This will allow DaedalusX64 to be truly multi-threaded.

This is going to take me a while to set up and I will continue working on non-ME related performance and bugs in the mean time, but this is my planned direction for DX64. Also in doing this I will be dropping OSX/Win32(I will restore this later for debugging on the PC)/XBOX support and this will become a PSP only version(temporarily until W32 is restored). However fixes and improvements we make will still be easily portable back to the original project(Strmnnrmn's -N64 branch). As I do this I will be rewriting several parts on this app to ensure that we are getting the most out of it. We've lost some decent speed somewhere along the way. Some titles in interpreter mode have seen over 20% drop in speed. This should also simplify the source code and have our processing going closer to the original N64 in the way it operates. I will also be commenting the source to make it easier for future developers to work on.

My first step is to learn about the ME and it's limitations. I have some ME test code I've been playing with and know some of the limitations of the ME now. Once the ME has been called:

1. CPU speed can't be changed we'll be stuck at 333Mhz (Not a problem)
2. Standby mode will not work. Not sure if this is a limitation of the current PSP SDK libraries or a Hardware limitation. So, no standby mode for us with the CPU on the ME since it will be called with every ROM. This limitation may only be temporary, but I have no clue. If it's permanent sorry to those that like to use standby.

First I will be rebuilding the main app. Now this part will be platform specific and be written for only the PSP to begin with. This means the DaedalusPSP project will still exist and the whole app won't become one project. The DaedalusW32 will be recreated later. This main app's purpose is only to set-up the environment variables, the user interface for ROM/Option Selection, and detection of system capabilities (AKA Phat/Slim/3K). I plan to look at the current GUI bugs at this time as well. Once a ROM is selected and "Start Emulation" is clicked, this is where things will be different. The system will then create the primary thread manager/memory manager first and then create the individual threads for each component as needed on the proper CPU based on code flags. Then the thread manager will act as the I/O interface between the individual parts so they can all be running asynchronously. This is what Howard was trying to accomplish, but I'm not sure it's possible under the current code. Once I have all the different threads running on the SP then I will push the CPU to the ME with the encapsulation already in place at that point I can set-up whether or not to use the ME as a flag so we can still debug CPU/Audio issues by using or not using the ME in Debug Mode as needed.

I also plan to rework the batch test and profiling code to help us further and last but not least add a wolf fence technique to the memory management to prevent leaks. As I work I will be bringing each piece back in slowly in small parts. Grazz will be helping with this and debugging the individual parts as we go to get rid of as many of the current bugs as possible. This is all going to be done on a private SVN for the time being with small fixes being back ported to the current code for testing. This process will overall take several months. Afterward, we will be posting the Beta 3 Preview in the Testers Section for final debugging. Then it will be finalized, committed and published as Beta 3.

The end result as compared to current Revs...
In Release Mode DX64 should be much faster and more stable. However the Debug version may be slightly slower when ME CPU processing is off than currently. This will be due to the thread manager overhead.

This is what's currently happening and my current plans. 99% of my other work on the project has been put on hold, but I think this is definitely worth while to pursue.