Tuesday, July 31, 2007

Seeing King Philip

Last week a few Lindens attended OSCON 2007 and yesterday, through the sldev mailing list, I received a link to a video of Philip Rosendale giving a keynote speech there. This was the first time I actually saw Philip and it was an good way of getting an impression of who he is.

Personally I'd say the speech itself was nothing that will be remembered in the great book of keynotes. It is certainly less entertaining than seeing Steve Jobs live, but he seems to be a likable dude. If nothing else, listening to the guy who rules this world adds a few interesting pieces to the bigger picture.

Reading between the lines and catching some of the finer points though, got me into an odd state of mind. It would be hard to describe what it is that leaves me less thrilled than before about the open source part of SecondLife, although this is a process which already started a few weeks back. Contributing free time to an commercial project always requires a balance and an acute and ongoing sense of the personal reasons for doing so. I really could not care less where in the imaginary food chain Linden Lab ends up, this has never been a concern for me, nor should it be. In fact I think that todays residents would benefit more from SL developing into a profitable market niche, rather than Linden Lab trying to fly to the moon in a soap box. But seeing this video really made me think and currently I am not sure about my future direction with the viewer at this point. But chances are (especially now that I have also seen the Voice First Look) that it may turn into more of an alternate viewer, rather than being a viewer which is merely a few patches ahead of the Lindens.

Anyway, if you're interested in what happens inside Linden Lab, I believe this video it is worth watching in the same way in which is was interesting to see Daniel Linden at a workshop back then when he blogged about keeping SecondLife safe together.

Grid woes and crashes

Most of you, if you are able to log on to SL at all, are probably experiencing more crashes than usual. I have been asked about the background a few times and figure the details will be interesting for many of you.

The crashes which you are currently seeing are mostly the result of unexpected conditions. With the grid and Linden network being in a woeful condition, probabilities are higher than usual that the viewer receives garbled or incomplete data from the grid.

Theoretically a program should be able to handle these situations gracefully. For example, if the login server is unavailable the program should display an error message (which my viewer does) and not just crash (which is still the "official" behavior).

Practically, it is almost impossible to foresee all the situations which can arise. It is possible to foresee some of the situations (like the login server being unavailable), but the irony is, that these situations are hard to reproduce, so they need to be cold coded (implemented without testing) and often the error handling code is causing as much problems as the erroneous situation itself. In the past I have fixed a few obvious ones, but in a software as complex as the viewer, it is simply impossible to plug all the holes.

So, if you are experiencing increased crashes at the moment, please don't worry that it might be something in your system. These are mostly caused by the viewer and there is nothing much to be done about them.

I would go bug hunting for those in a software that had a 98% quality level, just to put the icing on the cake. For my own software I am trying to actually plug all the holes because it is an entirely different situation, because I received the last crash report over six months ago. Or taking Adobe for example, I can't remember any Adobe program ever crashing on me under any condition.

But with the SL viewer, crashing under less than ideal conditions it is something which we will have to live with for a long time, because for many reasons, Linden Lab simply is not the type of company which will make quality and stability the first and utmost priority.

Thursday, July 26, 2007

About the viewer's memory usage

There's sometimes a bit of confusion about what amount of memory usage is normal for the viewer and how to determine how much it is using.

Of course it is not possible to exactly determine how much the viewer should be using, but here is a rule of thumb: 350MB + current texture memory + (5MB * number of avatars around you) + optional 100MB.

To determine the current texture memory, open the texture console (shift+ctrl+3) and look at the numbers in the upper left corner. The first number there tells you how much RAM the viewer is using for textures which are currently held in memory. The second is the maximum which the viewer will use for this purpose and depends on your video memory settings in preferences advanced graphics.

This first number will fluctuate wildly, especially for higher video memory settings and may even overshoot the 2nd number for a a minute or two, but both is normal due to new textures coming in and older ones are flushed out.

What you also need to look at is the processes tabs in the task manager. From the View menu choose Select Columns and make sure VM Size is visible. This column is the viewer's memory footprint and should roughly match the rule of thumb formula given above.

Example: You have your video memory set to 256MB and are at Bad Girls with about 20 avatars in proximity on the dance floor. The texture console shows 300/330MB in the corner. So you could/should expect a value of 350 + 300 + 100 MB => 750 MB in the VM Size column. Give or take 100 MB leeway because it is just a rule of thumb, so the viewer should not exceed 850MB in that situation.

I am posting this because there may still be memory lost in the program. But to determine or report incorrect behavior, it is first necessary to understand what is correct memory usage by design.

If your viewer is massively exceeding the numbers given above (say 1.1GB in the example situation) it would be interesting to find ways of making the viewer doing this. It may be related to specific activity or options or it can be basically everything (like streaming audio being turned on or whatever). But as long as the numbers match above rule and example, things are as they are intended to be.

Release: Nicholaz Edition 18e (experimental)

I was plowing through the source yesterday, which may lead to discovery of something that can cause memory fragmentation problems (unnecessary growth of memory and made a few experimental changes with what seemed like decent results. I was online with this code for some time, so it doesn't seem to create immediate problems, but I would like to hear feedback from those of you who observed memory growth and/or crashes in relation to large size textures.

If you want to play safe, please stick with 18d or earlier, but if you were experiencing these crashes (with such crash reports, I answered your emails with a reference to 2048-size textures), please use this version, look for improvements and make sure you send the .log files in case you crash again.

This time there is again no P4 variant, but my previous versions are of course still running fine.

Disclaimer: This viewer is unofficial and although I am doing my best to make it better than the release viewer, it may contain extra bugs and you will have to trust me that there is no malicious code in it. Please also understand that it runs on the main grid with your real Second Life account and inventory. Also read the disclaimer/header in the Install.txt in my archive.

Important: You will need Linden version 1.18.0.6 or higher installed before running my 18a-e versions.

If you choose to use this release, please have a look at the install instructions on my server and then download nicholaz-18e.zip from there. (Previous versions are still on the server in case you want or need to go back.)

PS: There is an in-world group for announcements of my releases: Search, Groups, "Nicholaz"

Wednesday, July 25, 2007

Release: Nicholaz Edition 18d

Here is a new edition of the Linden viewer. It has a few minor changes, among them one which will most likely fix the crash when emptying the inventory trash and another that should fix the multiple reloading of avatar skins when changing clothes or skins. See the change log and the associated JIRA issues for details.

Other than that, it contains a bit of extra debug code for those who are experiencing the crashes with 2048-size textures, so if in the past I have told you about these, please download this version and include the .log files in your crash reports.

This time there is no P4 variant, but my previous versions are of course still running fine.

Disclaimer: This viewer is unofficial and although I am doing my best to make it better than the release viewer, it may contain extra bugs and you will have to trust me that there is no malicious code in it. Please also understand that it runs on the main grid with your real Second Life account and inventory. Also read the disclaimer/header in the Install.txt in my archive.

Important: You will need Linden version 1.18.0.6 or higher installed before running my 18a, 18b, 18.c and 18d.

If you choose to use this release, please have a look at the install instructions on my server and then download nicholaz-18d.zip from there. (Previous versions are still on the server in case you want or need to go back.)

PS: There is an in-world group for announcements of my releases: Search, Groups, "Nicholaz"

Tuesday, July 24, 2007

Crash Report Triage

Bridie Linden posted an invitation to the User Interface Triage on the sldev mailing list. See here about more about who Bridie is and what she focuses on.

It will be held on Wednesdays 3PM at Bridie's house in Linden village.

The main focus will be to sort through crash reports on the public-JIRA and decide what will get imported to the Linden internal-JIRA , which usually is the next step toward an issue being addressed (it does not guarantee anything though). Meeting transcripts will be posted after the meeting on Bridie's wiki page.

You can help by attending the meeting and throw in your two Linden dollars. You can also contribute by pre-sorting the agenda before the meeting (which personally I find more effective). The usual routine for the latter is to sort the issues on the wiki page into categories or make comments or summaries for each point (see this older triage as an example for pre and post comments).

Saturday, July 21, 2007

Release: Nicholaz Edition 18c

It appears the last releases from Linden and myself were quite good. If seen occasional lag waves and packet loss on the grid but the viewers seems to do fine (and that includes progress on Able's project of a viewer which allows visual muting).

Over the last days I took care of a few minor annoyances, some of which have driven me mad at times (for example the fact, that double clicking a landmark opens a window). None of the these issues is critical in any way, so unless you're interested in a bit of UI tweaking, you'll be fine with whatever version you are currently running (oh, okay if you are still running Lindens 1.18.0.6, my version is about 40 patches ahead of theirs :-)). In any case, just see the change log for details.

Otherwise there now is an 18c version together with the P4 variant for Intel based computer (Pentium4, CoreDuo, Core2Duo ... right-click My Computer, Properties).

Disclaimer: This viewer is unofficial and although I am doing my best to make it better than the release viewer, it may contain extra bugs and you will have to trust me that there is no malicious code in it. Please also understand that it runs against the main grid with your real Second Life account. In this regard, it is similar to a First Look viewer. Please read and understand that link and also read the header in the Install.txt in my archive.

Important: You will need Linden version 1.18.0.6 or higher installed before running my 18a, 18b or 18.c.

If you choose to use this release, please have a look at the install instructions on my server and then download nicholaz-18c.zip from there. (Previous versions are still on the server in case you want or need to go back.)

PS: There is an in-world group for announcements of my releases: Search, Groups, "Nicholaz"

Thursday, July 19, 2007

Telltale Indicators

Slipping into my role as Dr. Nicholaz House, I'm looking over the patient's case sheet. (Well, his name is Gregory House, he walks on a cane and is looking ten times better than me, but I'll never pass the opportunity to use a metaphor, no matter how bad it is :-)).


Indicators

I've said a few times on the sldev mailing list that if you know how to read it, the official blog and (also the JIRA) are an extremely precise gauge of the state of the program. It just takes a bit of distance to look at the bigger picture and I put emphasis on the word just because I am very aware that this is quite hard to achieve when everybody is yelling at you. But assuming the right perspective, it is precise nonetheless.

There is a rule in Poker which says: If you look at the table and don't see the loser, it might be you. In a wild analogy I'd like share an lesson from my own school of hard knocks and say: If you look at the user's feedback and think people are not getting it, it might be you.


The Blog

A month ago, whenever a post was made with the comments open, the 100 slots filled up within hours if not minutes and the general tone ranged from acrid sarcasm to outright hate (with the occasional injection of reason or fanboithom). In a recent posts about "momentary" outage the comments died out at after about 25 and Torley's post today was well received. If those same texts were published a month ago, I can almost guarantee that there would have been a torch-carrying mob, having tar and feathers ready for any Linden who might be unfortunate or stupid enough to get into view.


JIRA

At the same time JIRA was practically exploding. I look at it daily and I remember that there were periods where I couldn't even keep track of the headlines there. Today when I visit, I can fully read every issue that comes up and most comments on existing issues.

There are still new reports there every day and this will never change. I know this from my own software. There is no software without bugs and there is no end to feature requests.

But you can assess the state of a software by the type of bugs and features which are reported or requested. The more obscure they become, the minor the details they address, the better the software. When I read the JIRA these days, it makes me smile. New issues are rare and specific, and the "fixed internally" list in the pipeline is impressive.

Admittedly there is still a lot of work to do and long standing problems (friends list, group notices, prims unlinking, ...) but these are the shit that happens and not the shit hitting the fan.


Psychology

What is interesting about SecondLife is that it elicits passion. If SecondLife was, as causal observers sometimes see it, just a fancy instant message platform and chatroom, people would not yell at the Lindens, the would just move on and talk to their friends on MSN, AIM, ICQ or IRC. Or speaking in another metaphor, a relationship isn't in real danger as long as the couple is yelling at each other, but it is about to end when they stop talking.

One of the expression most commonly heard in discussions about the problem was that the Lindens were "not getting it". If you want to convey a message and the other side is not getting it, you begin to speak louder, more pointedly and eventually you yell.

What I think the Lindens were not getting was how bad the actual user experience was at that time. I must say that I have some sympathy for them, because while I was spewing out patches I hardly spent time in-world and the SecondLife deteriorated from a life into a bunch of features and source code. It changed from an experience into a piece of software. In such a situation you need feedback from the users and with the users being a torch-mob there are probably only a few to get through to one and these may have their own agendas.


Timing is Everything

In today's Torley wrote an interesting line:

As we add aesthetic enhancements to Second Life in addition to much other work, we continue to think + feel these social dynamics despite misconceptions which don’t show “the big picture”.

I really like Torley and applaud her enthusiasm, but in this line, while I fully agree with it today, it still sounds a bit of "not getting it" what happened yesterday.

It is all a matter of timing and for correct timing one needs to understand what is happening out there. One or two months ago, there (hah!, metaphor alert) was a fire in the kitchen, the car had just broken down, there was hole in the roof with a storm approaching, there was water ankle high in the basement and hubby had just been told that they would let him go. Which in terms of timing is not exactly the right moment to suggest to buy a new stereo or to a change of wallpapers in the bedroom.

The events on the blog were the social dynamics and SecondLife is quite full of those.

But please don't get me wrong, I'm not scolding Torley, I'm just rambling. Her recent posts, her enthusiasm and efforts on the JIRA are adorable.

People are requesting minor UI changes on the JIRA already and even I am submitting patches about quirks rather than crashes. The fact that there was no mob on the blog today indicates that the timing is fine now and even I am beginning to look forward to WindLight. (Not that I'm overly enthusiastic, I have "grown up" under the old sky and am not too fond of the thought of never seeing it again, I will probably even make an effort to preserve it in my viewer edition).


Medical Outlook

I am sure that the next version (which my crystal ball tells me is in Q/A already), will have the number of crash reports go down by an order of magnitude compared to 1.16 or 1.17 and before long people will demand a new stereo or new wallpapers.

So, your (not always so) friendly doctor can retreat momentarily to his office, playing with his softball or watching a soap on his mini-TV while knowing that the patient is recovering.

Unless Linden Lab is doing something utterly stupid like trying to avoid rehab (which I don't expect), the worst is over.

Wednesday, July 18, 2007

Linden Open Source Awards

The Lindens announced their Open Source Awards on the blog today and many have nominated me in various categories. There are too many posts to say thanks individually there, so I'll do it here on the blog: Thanks everybody for the incredibly positive feedback.

I have said many times, that this feedback (here on the blog, via IM or email) is part of the driving force behind my submissions, besides the fact that I hate bugs and like the challenge in finding them.

There are two things I'd like to mention, though.

Despite the number of bugs, the SL viewer isn't a bad program. It is an incredible piece of software and has a size which goes beyond anything I have ever worked with. Most of the bugs are more like quirks, they usually have one particular reason and once being located, they are easy to fix. For a software with this level of complexity, which has been worked on for such a long time and by so many developers, the internal design is state of the art. This does not mean that it doesn't have it's messy areas, but I have seen smaller and less complex software which was much worse. I have seem projects, where you look inside and just think: "Okay, throw it all away and start over with it". With the Linden viewer this is not the case. On the inside the viewer is good, which is why it is fun to improve it. Inherently flawed software is impossible to fix. With the SL viewer, while it has it's bugs, they are easy to patch, which is the reason why I (and others) could fix so many of them in such a short time.

The other thing is that I was looking back over my contributions today. I submitted about 40 patches so far, but there are a few which I'm particularly proud or fond of:

  • VWR-364 was the first memory leak and it wasn't even really the Linden's fault. It took me a few days to find, but the hunt was thrilling. It took my particular field of expertise and I think it's the one with the biggest impact of all (VWR-1418 was equally big, but would have been easier to find).
  • VWR-870: is a fun story, because I accidentally fixed a long standing elusive problem. In a nutshell it was a combination of two quirks. One was a bug in one of the features hidden deep down in the viewer's client debug menus (regarding flexi prims), the second being the fact that the key combination to activate that particular (buggy) feature was also triggered on European keyboards when typing a closing square bracket. Which led to the conclusion that it's not a good idea to do programming (where you need the "]") when girls (with flexi hair or dresses) are around .
  • VWR-1436 happened on laggy conditions. It was a rather tricky technical issue and was the top crasher two weeks ago. I could not have found it without your help (through receiving those crash reports) and it is henceforth known as CSi Nicholaz.
  • VWR-1732 took me a bit to find (another tricky issue of the kind which C++ is prone of) but was nice teamwork: from Gellan Glenelg found the bug, Celierra Darling cofirmed repro (reproducing it), I located the underlying reason for the bug and offered a preliminary patch, Tofu Linden refined it (in a quite elegant way which makes my first version look embarrasingly ugly in comparison) and while testing I found another quirk with the issue (which was sheer luck).
Bottom line is that this project was and is a lot of fun spend time with. I don't regret a single hour invested there, even though there have been times of frustrations and moments when I was royally pissed. If I had to start over I again, I'd do it without hesitation.

The current Linden version is running better than anything they had since 1.13. There are currently over 100 bugs "fixed internally" and awaiting public release. There will still be server side issues, many of which have to do with the incredible growth in sim and user numbers, but I think version 1.18 will be remembered as a turning point in the history of the Second Life viewer.

Tuesday, July 17, 2007

Viewer Startup Options

There is a couple of startup options which seem to unknown to most (I accidentally discovered them when working with the source code), but which may be helpful nonetheless. They are appended to the target field in the icon which you use to start the viewer (right-click, properties, shortcut, target), e.g. "c:\program files\secondlife\nicholaz.exe" -autologin to change the behavior of the viewer at startup.


-login {first} {last} {password}
log in as a user
-autologin
log in as last saved user
-loginuri {URI}
login server and CGI script to use
-helperuri {URI}
helper web CGI prefix to use
-settings {filename}
specify the filename of a configuration file default is settings.xml
-setdefault {variable} {value}
specify the value of a particular configuration variable which can be overridden by settings.xml
-set {variable} {value}
specify the value of a particular configuration variable that overrides all other settings
-user {user_server_ip}
specify userserver in dotted quad
-god
log in as god if you have god access
-purge
delete files in cache
-safe
reset preferences, run in safe mode
-noutc
logs in local time, not UTC
-nothread
run vfs in single thread
-noinvlib
Do not request inventory library
-multiple
allow multiple viewers
-nomultiple
block multiple viewers
-ignorepixeldepth
ignore pixel depth settings
-cooperative [ms]
yield some idle time to local host
-skin
ui/branding skin folder to use
-noprobe
disable hardware probe
-noquicktime
disable QuickTime movies, speeds startup
-nopreload
don't preload UI images or sounds, speeds startup

UI Triage

Iridum Linden posted an invitation to the User Interface Triage on the sldev mailing list.

It will be held on the 17th, 10AM at Benjamin Linden's office in Linden village.

This is an opportunity to have a bit of impact on the way the Lindens are proceeding with some issues from the bug tracker. You can add things to the agenda and resolution of JIRA issues will be decided (for example there is one there at the moment with a patch from me which proposes a UI change and it will possibly decided on the triage if the behavior will be changed that way or not).

But the main focus is what will get imported to the Linden internal JIRA , which usually is the next step toward an issue being addressed (it does not guarantee anything though).

Do not expect too much immediate impact, but if you want to get your hands into the clay, at the moment this probably is as close as you can get to steering the Lindens, without being a programmer yourself or without spending major time on sorting out issues on the bug tracker.

If you're interested, look at the two links near the top of this post or see the notes from previoius triages.

Saturday, July 14, 2007

Release: Nicholaz Edition 18b and 18b-P4

The Linden Labs viewer turned out to be without major problems, only the grid (and electricity on the server side) seems to be slightly on the laggy side, at least I am seeing more packet loss than before Wednesday.

With this, I decided to make a new build and include a P4 optimized version this time. The changes compared to the Linden viewer are somewhere north of 30 patches, compared to 18a there are just a handful fixes of minor impact (please see the changes history for details).

With an Intel Pentium-4 or CoreDuo (right-click, Properties on My Computer) processor you will probably dowload the 18b-P4 instead of 18b, because it offers a bit of general optimization. Some is targeted at Pentium processors, but will work for AMDs as well, I just have no idea about the performance impact on these ... it might run a bit faster on these as well, or a bit slower.

Disclaimer: This viewer is unofficial and although I am doing my best to make it better than the release viewer, it may contain extra bugs and you will have to trust me that there is no malicious code in it. Please also understand that it runs against the main grid with your real Second Life account. In this regard, it is similar to a First Look viewer. Please read and understand that link and also read the header in the Install.txt in my archive.

Important: You will need Linden version 1.18.0.6 or higher installed before running my 18a or 18b.

If you like to use this release, please have a look at the install instructions on my server and then download nicholaz-18b.zip from there. (Previous versions are still on the server in case you want or need to go back.)

PS: There is an in-world group for announcements of my releases: Search, Groups, "Nicholaz"

Two more mysteries solved

In a recent blog post I said that when the viewer doesn't do what it should, it's not always the Linden's fault. Now, not that it's unnatural to blame the viewer, because with problems everybody looks in the direction which is most likely to yield results and the SL viewer was (and sure is) to blame often enough to look there first.

But sometimes problems lie outside. In that blog post (and subsequent comments), there were two people with broken fans on the graphics board.

I was also in email contact with two other users since a few days because they had mysterious malfunctions.

One had unexplainable memory growth. The swap file was growing, the viewer eventually slowing down as with bad memory leaks, but none of the programs in the task manager's process list indicated any excess memory consumption.

The other had occurrences where he would crash on login. He said that he had to reset his router/DSL modem to get it going again. I looked into the crash and found a nice Linden bug, but the problem remained (just without the viewer crashing, instead it just gave a proper error message).

The solutions in both cases are quite specific (and would probably be worth going to the Linden support database): The memory growth was a video board with shared system and video memory, in other word, using more video memory resulted in less system memory. The login problems resulted from a internet connection device (NetGear rp614v3), which when having enabled a feature to prevent DOS attacks, locked up https connections.

Friday, July 13, 2007

Feeling Good

I'm in a fine mood tonight. I've just submitted three nice patches. Most of you will never notice the issues or the improvement and one is internal only anyway, but they were really lovely. Easy to reproduce, not too hard to find, no urgent matters. Best of all, they were fixed with just a few lines of code (two even with a single line).

Reminds me of my earliest contributions, when I did it for the pure fun and when the bug hunt put me in a nice meditative state.

The dust is really settling and I plan to spend more time in-world again. If you know any nice places to visit, please post your favorite places here as a comments or drop me a landmark.

Taking back the View

A few weeks ago I saw an issue about ad-farms. While reading through it, I picked up an idea from a comment there and offered it as a separate issue (it's really not my own idea, I just promoted it a bit). Subsequently it has drawn a some attention and discussion on JIRA and on the sldev mailing list.

A fellow developer (Able Whitman, the guy who brought you the changed dialog to avoid the debit spoofing) started working on it and is now offering an experimental viewer which let's you visually mute prims or objects on parcels.

I do not own land, so it is essentially a non-issue to me, but I did test an early version. Able has done a terrific job there (he has a persistence and dedication like an alligator's jaw) and the effect of applying this function on undesired objects is quite profound.

The viewer comes with a full installer, includes some of my patches and will not interfere with your Linden installation or my edition.

So, if you are in an adventurous mood, are pestered by ads around the place where you live or if you just want to help out a fine coder with some feedback, please click here to find out more.

Thursday, July 12, 2007

Release: Nicholaz Edition 18a

The Linden Labs viewer seems to be reasonably stable and there's no indication of an imminent update, so I decided to go forward and do an 18a (a P4 optimized edition will probably follow in a day or two, watch for updates to this blog post).

There seem to issues with textures and some users still experience strange growth of virtual memory swap files. These sound tricky to nail and will probably take time to be sorted out. If you have any of those issues, please consider to provide details under VWR-1694 and VWR-578 and/or post observations as comments here.

The differences between my viewer and the Linden version are listed in the change log. The gap isn't as wide as it used to be, mostly just my usual patches (particles, memory usage, blurry textures on low end graphics boards, the odd crash), plus some interesting ones from the backlog on the Patch Sematary (tribute to Steven King). Half of them are by other open source contributors, so the half the credit should go to my fellow developers out there. (All JIRA patches are listed at the bottom of the change log by JIRA numbers and a semi descriptive file name and are also included in the source archive as patch files.)

Disclaimer: This viewer is unofficial and although I am doing my best to make it better than the release viewer, it may contain extra bugs and you will have to trust me that there is no malicious code in it. Please also understand that it runs against the main grid with your real Second Life account. In this regard, it is similar to a First Look viewer. Please read and understand that link and also read the header in the Install.txt in my archive.

Important: You will need Linden version 1.18.0.6 or higher installed before running my 18a.

If you like to test and use this release, please have a look at the install instructions on my server and then download nicholaz-18a.zip from there.

PS: There is an in-world group for announcements of my releases: Search, Groups, "Nicholaz"

Wednesday, July 11, 2007

Linden Viewer 1.18.0.6 (2x Updated)

The Linden viewer 1.18.0.6 is out and the grid is running again.

No word about the source yet, so don't hold your breath waiting for 18a, it won't probably happen today.

If you are online with their new viewer, please post problems or experience as comments to this post. If there is something wrong, I'll be knowing what to look for.

Meanwhile, enjoy your second time there ... :-)


Update: Looks like you'll be enjoying blurry textures or gray objects mostly at the moment (at least in new places). Source code has not been released yet and I don't think there will be an 18a any time soon (most likely earliest date for that will be Friday, see comments to this post).


Update #2: Things seem to have settled down. Textures load fine here so far and also everything else seems to be working. So I guess I was a bit too pessimistic this time. Kudos Lindens!

Source code was also released, so I'm putting up an 18a on the server now for early adopters. Proper announcement will follow tomorrow (meanwhile see changes.txt on the server). Put comments about 18a also here on this blog post please.

Lindens of the Year

There's currently a contest running on JIRA about open source contributors of the year. Lindens are excluded, so despite my recent dissatisfaction with some Linden'ish proceedings, here are my Linden of the Year nominees.

Best Feature
Qarl Linden: The guy who implemented sculpties, which is a feature with massive impact. From what I heard he did most of it in his spare time. For a feature as extensive as this one, it's implementation met the highest standards of code quality: almost no bugs in the feature and zero adverse side effects outside the feature. (Quarl also gets a honorary mention in the easter egg hunt of the year category for what came prior to the announcement; if you missed it, read the blog posts and comments on that link from bottom to top.)

Best Bug Hunter
Anonymous Linden: There has been a lot of bashing about Lindens not fixing bugs, but please take a moment and have a look at the releasenotes.txt file in your c:\program files\secondlife folder).

Best Community Organizer
Torley Linden: The Linden who spends most visible time on the bug tracker and who is the one most caring about what the people out there do and say. (Torley also gets a honorary mention in the psychedelic signature of the year category.)

Best Patch Importer
Sardonyx Linden: He's the one who imported most patches from JIRA (and spotted a major memory leak just from the source)

Best Linden Blog Post
Joshua Linden: Today was a little more exciting than I like my Tuesdays, but .... I'll remember that for a long time. Friendly, no bullshit, just in time and the right thing announced.

Some like it hot ... NOT!

No, this post is not about sex in SL :-)

Here is a funny one. I was emailing with a user who had mysterious crashes which locked up the whole computer. Not the viewer crashing, but the system doing an auto reboot during Second Life, totally random and with no specific in-world action (none that could be determined at least).

Hard lockups and cold reboots are among the worst things to debug, because it is basically a mystery with no corpse. The interesting issue is, that with modern operating systems, an application usually is not able to cause these. With blue screens, system lockups and such, there is in 99% of the cases a driver or deeper operating system component involved.

Sure, Second Life seems to "cause" a fair amount of those. For example on my work PC with Intel on board graphics, I had frequent lockups before I bought an egg fryer (a Nvidia 6800 with passive cooler). The my current view of this is, that the viewer is pushing GL to the limits (and sometimes beyond) and some drivers seem to handle that less than gracefully, thus in these cases I tend to agree with the Lindens, blaming these on the drivers. (As a side note, for my business I develop communication software and our software so far has crashed most of the 3rd party com port drivers, because we also push these to the limits.)

When it comes to drivers, I am quite merciless. No matter what an application does, a driver should slap the application's fingers, it can crash the application in a ball of bytes, fire and smoke. But a driver has to catch the most insane events and handle them without crashing itself and the system. Period.

However, the reason why I am saying this is, that with events that go beyond crashes it may be prudent to look outside the viewer for a cure. Checking memory, trying different graphics drivers, check software and tools installed on the system.

This is what I recommended to this particular user, but I seem to have also had a lucid moment and recommended to check something else.

He said he'd do a CSi on his system and here is what he emailed me yesterday night:

... the fan on the display card had stopped working!

Some like it hot, but hardware ... NOT! :-)

Tuesday, July 10, 2007

CSI Nicholaz (Episode 2)

I recently coined the term CSi Nicholaz for bug hunting with help of crash logs. Yesterday Soft Linden took the effort to post the latest Linden crash stats. Thanks to your help I know most of these quite well (the info posted there is just a fraction of what I get through your dumps) and am happy to say that the bugger caught in the last CSi Nicholaz is the first crash on the current Linden list (they do not yet have that fix implemented).

The bad news is that with each crash fixed that way, the remaining ones will be more and more mysterious. You first catch the dumb criminals (despite the occasional lucky case) and when when those are behind barred windows the ones running free are harder to get.

Since CSi Nicholaz I think I have looked at about 20 - 30 dumps without much success, but yesterday I received one which was a hot lead. I talked to the witness (well, the user who sent the dump), he was quite insightful and could provide further clues, so today I was able to reproduce the problem and that was the end of the little bugger.

The bug is like most these days, an error in error handling. The viewer today runs reasonable stable when nothing unexpected happens, but the unexpected events come in many forms and the error handling of these conditions is hard to test, so erroneous error handling isn't uncommong in software (the first CSi was of the same class, as well as another crash and at least five minor leaklets).

This one refers to LLUserAuth::authenticate with appears twice in the list of the current top ten crashers.

Yay!

Going the full circle

I just noticed that I filed my first patch on May 11th, which was, almost to the day, two months ago. I had downloaded the source the day before and had just managed to compile it into a viewer. The problem had been simple, I was able to reproduce it and the patch was just a single line of code changed in the hundreds of thousands of lines in SL.

I have lost track about the patches I submitted since then but I did fix another bug yesterday which is similar to that very first one in that it was solid and easy to reproduce, did take me about the same time to locate and essentially also requires just one line of code to fix.

So in a weird way I have returned to where I started. I know a lot more about the viewer and Linden Lab than I did when I started. So this may be a good opportunity for a recap.

Back then I was already aware of the memory leak and remember mentioning to a friend how cool it would be to be the person who finds and fixes that.

Well, as you are probably aware, it turned out that I am the person who found and fixed it. Not only that, but also another leak of equal proportions, together with a dozen minor ones, together with a hand full of crashes. Among these crashes is, as I found out yesterday, one which is currently the number 1 among those arriving through the crash logger at Linden Labs. That is, number one now after the previous top crashers, which were caused by the memory leaks, are gone.

I've sure came a long way with this project, but besides being proud of what I achieved, I am quite disillusioned about the whole project.

Yesterday I became aware of an problem which was introduced with 1.17.3. This problem made mequite acutely aware how the Linden way and my way of dealing with software are quite different, if not utterly incompatible. I'll spare you the technical details (you will find them on the sldev mailing list archive under the subject of "SSE / ouch"), but this issue is, as they say, the drop which spills the glass.

There will be a 1.18. release tomorrow, which renders previous versions useless and which the Lindens are very excited about. A month ago I would have been quite optimistic about this release as well, but as of now, all I can say is that I'm not holding my breath.

I plan to continue to offer improved editions, there seem to be some of my changes which people like or need, and with the current backlog in the Patch Sematary (which incidentally is about as large as two months ago) it will also be easy to offer an improved experience for users.

There is one thing that will change however. In the past I have mostly focused on getting my stuff into the official Linden viewer. I recognized yesterday, that this is the part which caused the most frustration and I have already given up efforts in some areas which are geared towards the Linden edition.

Instead I will simply concentrate on my own builds and will try to provide a good Second Experience to those who are using it (yes, that's you :-)).

Sunday, July 8, 2007

Repost: Tweaking Performance

Here's a link to an earlier post about improving performance for users of the blog who may have missed it. The part about the video graphics memory is still relevant, especially if you want to go back to the official viewer or if you are running multiple concurrent viewers.

Saturday, July 7, 2007

Release: Nicholaz Edition 17i and 17i-P4

I just made a new release 17i to fix the stupid bug I made regarding the "[person] is typing ..." message in the IM window.

As promised I also made a P4 version, which contains exactly the same source code, but is made with a few optimization options enabled in the compiler. It may or most likely run a bit faster (these smaller tweaks are always hard to evaluate) on Pentium 4 processors. On other processors I am not sure if it has negative impact ... feel free to choose either of the two versions as you see fit.

In all other respects, this release is identical to 17h minus the bug in the IM window, so for more details please continue to read here.

A Linden-like bug in 17h

I have a bug in my viewer. That's not a problem in itself, but what is worrysome is that it is very Linden like.

The patch which I made for the IM window ("[other person] is typing ...") backfired in a big way. I am sure I tested it, but yet it now just made things worse than they were before (you'll always see the "[other] is typing ..." message now and I have yet to figure out who this slipped through after testing).

Sure, it's just a cosmetic issue, but you can expect to see a fix soon.

Friday, July 6, 2007

Release: Nicholaz Edition 17h

Now that Linden Labs has released version 1.17.3 and I am also ready for my next version.

Apparently the Lindens managed to apply the leak fixes eventually, so there will not be as much difference as there used to be between the official viewer and mine. I see this a good thing because people will now be able to have a decent experience in second life and this is the reason why I started this project in the first place.

Currently the grid seems to be in good health also, so hopefully everybody (users of the new Linden version and users of all my versions) will hopefully have a nice Second Weekend before the update fun starts again next Wednesday. :-)

The Lindens did not release the source code for the new viewer yet, but this is a non issue because it was optional and most fixes are in my viewer already anyway. In other words, Nicholaz Edition 17h is simply the next step after 17g (which was based on 1.17.2).

However, even with the leak fixed there should be still a a few areas where I am ahead of the Lindens and this is as I think it is supposed to be. With that I mean that alternate builds like mine should be there for small and specific tweaks, but in an open source environment they should not be able to run circles around the official version for weeks ongoing.

I have lost a bit track (and also interest) of what fixes they have in their viewer and which not, because the backlog in application of patches seems to be growing again lately and I am getting tired of checking these, especially with the bug tracker lately acting more erratically than before.

But here is a rough overview what you can expect from 17h compared to the official 1.17.3 build:
  • slightly better performance (btw, there will be an Pentium4 optimized 17h release soon)
  • less crashes (I think there are four or five possible crashing bugs fixed)
  • more consistent particle generation
  • better memory management on systems with not enough memory (less 750MB)
  • better textures on graphic boards with less than 64MB video memory
  • various minor user interface issues (e.g. "... is typing" in IMs)
  • some keyboard shortcuts added (rebake texture, disable camera constraints) or changed (fix issues with European keyboards)
  • the new "Are you sure you want to quit the program?" dialog is not in my viewer
Click here for the full list of changes vs. official Linden viewer (see the entries for 17g and 17h there).

There is one crash in particular which I hope to have fixed but only time will tell, so I will need your crash dumps to see if that bug will appear again there or not (see the AboutCrashes.txt file for details).

Disclaimer: This viewer is unofficial and although I am doing my best to make it better than the release viewer, it may contain extra bugs and you will have to trust me that there is no malicious code in it. Please also understand that it runs against the main grid with your real Second Life account. In this regard, it is similar to a First Look viewer. Please read and understand that link and also read the header in the Install.txt in my archive.

Important: You will need Linden version 1.17.2 or higher installed before running my 17h.

As always with optional releases, if something goes wrong, just go back to one of my previous versions (17e or 17g2 are both good).

If you like to test and use this release, please have a look at the install instructions on my server and then download nicholaz17h.zip from there.

PS: There is an in-world group for announcements of my releases: Search, Groups, "Nicholaz"

Linden Labs Viewer 1.17.3 (Updated)

I just noticed that yesterday evening Linden Labs released an 1.17.3 viewer. They did not give us the source code yet, and the feedback from the comments on their blog seem to be mixed so far.

I'm especially worried about people saying that this viewer is even slower than the previous version 1.17.2, where technically it should now be able to run circles around his older brother, because of the new leak being fixed.

However, I will still proceed as planned, releasing my 17h later today, based on 1.17.2 and will be patient and wait until we get the source code and take my time to check it and to see what the feedback about it will be.

Meanwhile, if you download and try 1.17.3, please let me know on the blog comments here what your experiences are.

Update: Judging from the fact that there are still open slots for comments on the Linden blog, it seems that their new version does not have any serious problems. There may to be a small problem on some machines while installing (I did not encounter anything like that myself) and some say it is slower than before.

I did a test run myself for about 30 minutes now. It appears that there is no major memory leak, memory usage stayed within the expected and I did not observe any dramatic speed issues. So from my point of view, this is a viewer which is well usable and Linden Labs eventually arrived at a point which I expected for version 1.17.0 or 1.17.1 (see my older posts about their versions).

This is especially good news for users of Apple/Mac computers, because it is the first time they can expect to run leak free.

Dr. House (Part 3)

The patient is leaving the hospital after enduring extensive treatment, but as it turned out, most of the symptoms (being able to reproduce the error consistently) were a result of a very early medical procedure.

This is where the Dr. House analogy ends. In the TV series at the end of an episode the patients either leave the clinic fully cured or sometimes they die. With this particular bug however, only time will tell if the treatment actually fixes the problem or if it was merely a cosmetic procedure, because it still is elusive and I will have to see if I receive new crash dumps after the next version which break the program in this particular area.

Bottom line: I administered a change which definitely makes sense in terms of cleaning up the code and with some luck, it will actually fix the bug I was hunting. Without the wild goose chase, I would not have made the necessary efforts, to address this change, so in a way it's either a good thing (cleaning up something in the code) or a very good thing (fixing a bug while cleaning up some source).

There will be a 17h (this will be the Dr. 17House Edition) soon, most likely tomorrow, offering nice small fix, which I am sure everybody will notice.

So have a nice evening, thanks for watching and tune in again when the "Mad Patcher is in da house" :-)

Thursday, July 5, 2007

Dr. House (Part 2)

After writing that I felt a bit like Dr. House I am now sure that I teleported into the series. If you watch Dr. House, you have probably noticed that many of the episodes follow the same dramaturgy: Patient arrives at the hostpital in a bad state, the initial ideas about the the illness have been proven wrong and after the return from the first commercial break a new assumption is made which fits quite well and the new drug is administered.

This is what happened this morning. The symptoms were evaluated, the reason seemed clear, treatment was started and I went to work (commercial break). The bad news is that it did not help.

With the TV series however, you know how long it will take and that there needs to be another twist because they will have a second intermission, trying to sell you toothpaste, cars or health insurance.

Incidentally, I'll go to the movies in an hour. I now seriously hope that this case will follow the same pattern as on TV, which means that after the break something totally unrelated will happen, which will spark an idea, I'll make the mental connection and everything will be fine.

Update: But then we find, that the patient aquired most of his symptoms in the hospital due to the very first treatment.

So, stay tuned, we will continue after a short break ...

(continued)

Dr. House

The compiler is currently running a full rebuild (which usually takes about 20 minutes), JIRA is down, there's nothing on the mailing list, so I have a bit of time (I won't mention that I took the morning off from work).

What the compiler currently does, will hopefully be a huge step forward in terms of stability of the viewer. I have one crash that has been very high up on my Most Wanted list (it is quite high on the Linden top 10 crashes also). I have aquired a nice collection of crash dumps coming from this problem but never had it myself. Using my CSi Nicholaz metaphor, these dumps are all pictures of a dead program with no visible reason. It's like sending a blood analysis to the doctor and then the doctor says: "Well, it all looks fine" and you think "Umm yes, but you know, the person is dead."

My favorite TV show is Dr. House and each time I watch it, I'm amazed how similar their approach to diagnosis is to what I do. Dr. House is a programmer, he's just debugging people.

Yesterday night when making some changes for the 17h release my viewer died. I need to say that I did not see a single crash in a few days, so I was sort of delighted (I guess I'm probably the only person on the planet who likes it when the SL browser crashes). Having the program crash in my debugger is like being able to perform a full autopsy rather than just seeing a blood sample, but the program still looked good. Except that it was dead.

I started the full Dr. House routine. MRI, CAT scan, tissue samples, everything. Or in programming terms: Heap check, object contents, logs, memory dumps, following pointer chains . Result: the program should not be dead. But it was. At that point, I wasn't as delighted anymore, because this is the worst possible situation. A crash that rarely happens, with no apparent reason visible in the debugger. Duh!

I did a debug build (this is a special form of compile with terrible performance but maximum information). And I was lucky. Well, sort of. The crash happened again. The program still looked good, but it was reproduceable. So I went from autopsy to the diagnosis because I could practically watch the program dying (this is the point where I prefer programming over medicine). That was 2:30AM in the morning and I thought "there goes my sleep" (and a pack of cigarettes).

Oh, my compile in background is through, will tell the rest of the story later. Don't hold your breath, it may take some time to dig through the results and I'll need to go to the office too.

But I think there is a light at the end of the tunnel. Hopefully this time it will not be the kind of light, which people are reporting from near death experiences, and I also hope it's not an oncoming train :-)

(continued)

Wednesday, July 4, 2007

Download Stats

I am no fan of statistics and I am gauging the success of my viewer more through the feedback than by the number of downloads, but today I had a look at the logs for the server where the archives are hosted for download and was quite surprised by the results, so I dug up some

I had idea to offer my own edition of the SL viewer some time after I found the memory leak, when Linden Labs made no visible effort to fix it soon. I put the viewer which I was using for myself on a server and told a few friends about it. I said that it was okay to give it to friends, but asked them to otherwise not make much fuzz about it. This was around on May 19th (one and a half and month ago).

That version (just Nicholaz Edition or nicholaz-b.zip) was downloaded 24 times. The first version mentioned on this blog was 16d (May 28th). I don't remember exactly why I began to make it public in the first place, but over all it was downloaded 40 times.

These aren't impressive figures, especially since the first people sure were downloading that 16d version also, but making 40 people happy with a bit of coding isn't a bad thing either. I expected the number of users to stay around this figure, because the Lindens said they would fix the leak, which is what I wanted and at the same time would not have given people much reason to jump through the hoops of downloading and using a viewer made by a rogue coder (as someone called me on a forum entry).

Things turned out a bit different than I thought though. The mandatory 1.17.0 release did not fix the big leak and I made 17a head over heels. Then the Lindens announced that they would fix it with 1.17.1 and did, but at the same time introduced a new one of equal proportions. Sardonyx Linden mentioned this blog on a post on the official Linden blog and the number of members in the SL group for the announcement of new versions made a small jump (I think it went up to 60).

Last time I checked the group (a few days ago) it had a bit over 100 members, which I thought was about the number of users, but when I checked the download numbers yesterday, I was in for a surprise.

Version 17e (based on Linden 1.17.0) was downloaded 920 times. Version 17g (which was just released last Sunday based on 1.17.2) now has a download count of 520. Certainly many overlap there, but I guess it is safe to assume, that currently about 1000 people or more are using this alternate viewer.

As said, I don't judge my success by download numbers, but helping a thousand people to have a better second life sure makes me happy.

Sunday, July 1, 2007

Release: Nicholaz Edition 17g2 (updated)

I have been working on Linden Lab's 1.17.2 version since it came out and I am now ready for a Nicholaz Edition based on it.

I had high hopes for this Linden release but it seems that time was not ripe for a solid viewer by them.

The lucky thing however was, that they postponed the 1.18 release, so everybody could just continue to use my 17e browser, which is still running a lot better than anything the Lindens released in months and which gave me enough room to use some extra care with my new version 17g (in case you're wondering, 17f was for my own use while developing).

However, with the extra time spent on the version, I hope to bring you a viewer that is better than ever. In order to do so, I also included a couple of patches by other contributors and I am very happy to see the open source community offering more improvements than ever.

The highlights since version 17e are four fixed crashes:
Plus the improvements which came from Linden Lab 1.17.2 minus the un-improvements which were added there. Click here for the full list of changes vs. official Linden viewer.

When using 17g you can expect to crash a bit less than with 17e with the extra benefit of having a few nice changes to the user interface. And you can expect to run a LOT better than the official viewer.

Of course even with my versions there is a risk of breaking something, so 17e is still on the server and and excellent choice to fall back to.

Update: There was actually something broken in 17g. Some users experienced problems with blurry textures, which should be fixed in version 17g2 now.

Disclaimer: This viewer is unofficial and although I am doing my best to make it better than the release viewer, it may contain extra bugs and you will have to trust me that there is no malicious code in it. Please also understand that it runs against the main grid with your real Second Life account. In this regard, it is similar to a First Look viewer. Please read and understand that link and also read the header in the Install.txt in my archive.

Important: You will need to install Linden 1.17.2 before running 17g.

So, if you like to test and use this release, have a look at the install instructions on my server and then download nicholaz17g2.zip from there.

PS: There is an in-world group for announcements of my releases: Search, Groups, "Nicholaz"