Sunday, November 25, 2007

Long Road Behind, Long Road Ahead

For those of you who missed it, Philip recently made one of his rare appearances on the blog, talking about about Linden Lab's new mission statement which now reads "To connect everyone to an online world that improves the human condition" and SecondLife's future in general.

His post is quite ironical in two ways.

One is the the mission statement's use of the word connect. The post was made on a day where SL was unreachable for almost 24hours for many residents and if you look at the Linden blog today and scroll back, you'll see a series of posts about connection problems every single day between the 17th and 24th of November. In fact today is another rainy Sunday on which I wanted to toy a bit with the source and even spend some time in-world, but the only results I got was were "Login failed", "Site is taking too long to respond" and "503 Service not available".

The other bit of irony in it is the title of the post: "Long Road Behind, Long Road Ahead." Philip certainly is a visionary and his vision is surely what made SecondLife possible in the first place. But it occurred to me recently, that a lot of friction and frustration between Linden Lab and the SecondLife residents, may be caused through a difference in focus.

I do not want to go into the spiritual or philosophical arena too much, but there are many schools of thought, who are beginning to discover, that happiness is rooted in the here and now, while constant focus on the future or past tends to create angst, frustration and dissatisfaction.

SL Residents naturally are focusing on their experience in-world right here right now. Personal tolerance levels obviously differ, but nobody can live on a vision of a better tomorrow, if tomorrow never comes. On the other hand, over the last months if not year, Linden Lab seems to have worked based on the idea, that at least this next thing, is essentially required to keep them ahead of the competition (Philip quotes Voice in his post and Torley currently and tirelessly beats the drum about how Windlight will improve the depth of the experience).

Looking back over my experiences with SecondLife as a developer and pondering Philip's post made me aware how different this kind of focus is inside and outside LL. Probably the most prominent example of this is the status of issues on the bug tracker when they are marked as fixed internally. For a Linden developer such an issue is dealt with and finished, on the outside it means it's still unfixed here and now and circling in an orbit of Linden Lab's internal procedures, slowly descending towards impact on the official version. Looking at the JIRA and searching for issues marked fixed internally currently yields 159 results for the viewer. Technically they are marked as resolved, which of course reflects the Linden view (or the view of the devs who go to office hours), but it is obvious, that the same (the issue being resolved) is not true for residents.

Don't get me wrong, the developer's standpoint is legitimate, it is their view and experience that the problem has been fixed, but there ought to be someone understanding and tending to the gap between developers' and residents' point of focus and experience. Torley might be this person, feeling the gap (welll, when he's not all excited about Windlight), but regrettably his influence on the whole process seems to be close to zero (invariably, when he posts on JIRA that he's forwarding some issue or finding to the devs, subsequently nothing much is happening) . Look at MISC-124 on the JIRA if you need an example.

I was recently sent a transcript of a Linden office hour, where an interesting statement was made by a Linden: "... but my understanding is that the [Nicholaz] fixes that haven't been incorporated mask the real problem rather than address it."

I don't want to go into details about my individual patches here, but what my changes always do is to address a problem quickly or at least increasing the chances to address it. I think I never submitted a patch which made things worse. Even if they had the potential to push a crash towards another place in the source (by adding a check to make the program flow pass a crash point but with the possibility of failing later) they never made the source less readable or less reliable. However, this is somewhat besides the point, because even if they were nothing but band aids, they had an immediate effect, which obviously is felt by a lot of people.

But again, it mostly outlines the difference in focus between residents and LL.

Here is an example: Two of the first patches I did were submitted in mid May. One is a minor issue of no real importance (layout of the texture debug window), the other had some major impact on memory use. Both are a nice example of what I am trying to say. When I submitted them, someone inside LL said that the issues would be addressed in a different way with a more general redesign in a later release. Now (over six months later), one of the two is still open, the other is fixed internally. I don't really care about what's the state of these on the JIRA, but today both still affect end users in exactly the same way as six months ago in the official version and release candidates.

Another example like this is the Chatterbox. The Lindens have said that they will address the problem. They way I understand their line of reasoning (although this purely speculation), it means to put (off) the solution into a basket of general GUI redesign (a project which seems to be in Linden orbit for some time now).

In a blog post a week ago, I was answering a question by JayR about the difference in quality of my viewers in comarison to the Linden ones. I tried to provide an answer in that post, but I now think I missed an important point there. A good part of the answer, as I see it now, is "... because I focus on the here and now."

How long does it take Linden Lab to replace a broken light socket? About six months. Because there are currently plans to completely renew the electricity in the house anyway, which you will understand takes some time, thank you for your patience.

I think what many residents are tired of, is hearing the word later. Maybe not all, but certainly many who are using my viewers. Living based on visions works for some time, but it is part of the human nature, that experience happens in the Now. Which incidentally is the part which is missing from Philip's headline.

A developer can draw his now-experience from designing and implementing visions, but users are living an entirely different perspective. Successful companies in the Web 2.0 area are giving their users/clients meaningful and satisfying experiences, which then may serve as a solid platform to add the excitement of visions.

I don't know if I'm right in all this, but a blog post with ramblings like this is what you get on a rainy Sunday with login failed and 503 service unavailable :-)

26 comments:

Ram said...

Thanks Nicolaz. If Linden Lab does not understand this perspective, then we are in trouble. It took me a long while to understand that what matters is the NOW, always.

bhw2249 said...

Just figured I would note, they tend to write these posts days ahead of time even weeks, and then let them "expire" or "go live" at a certain time/date, so my guess would be this was timed with the US holiday and no one with high enough authority was going to delay it being released because of operational issues.

Anonymous said...

"Looking at the JIRA and searching for issues marked fixed internally currently yields 159 results for the viewer."

Please don't take this to mean anything. I resolved all the old windlight beta bugs as "fixed internally" to get them out of the way so that we don't confuse them with the new windlight bugs. (Yes I did ask some Lindens before doing this).

It concerns me that people point to this number, when probably 100 of the bugs in question are simply old windlight bugs that were resolved because there was a large chance they were no longer valid.

-Gigs Taggart

Ordinal Malaprop said...

Hah, yes, the "resolved" issue has really begun to irritate me; I did submit a JIRA bug (WEB-380) that issues should not be marked "resolved" until the fix has made it into a final release (I look forward to it being marked "resolved" with comment "will be added to planned upgrade of JIRA software" :) ).

The thing is that issues are with the client, or the server, not the code. People submit issues based on the behaviour of Second Life. Therefore those issues are not actually resolved until they no longer appear in Second Life. Obviously one needs to fix the code to fix the issue (unless it is a hardware issue I suppose) but incorporating the fix into viewer releases is part of the process of fixing it.

Ancient bugs that have perfectly valid fixes in existence and have done for ages - many incorporated into your own client, and none of which have made it into an official client version, drive me potty. As a content developer I am unable to use alternative viewers to design and test with, as potential customers and users and clients will almost certainly be using the official one. (I could only do otherwise in very occasional cases - say something like the ESC CSI project, where users will be using a specialised client - and I haven't been involved with any of those yet.) And as a Second Life resident I am simply annoyed that I would have to hunt down alternative clients to solve issues for which there are already fixes and have been for ages.

Alexandra Rucker said...

Nicholaz:
THANK YOU for putting that in words. It does seem to explain a lot of what's been going on.

Anonymous:
Old bugs are still bugs. Knowing that some have been marked that way just to "get them out of the way" .... begs the question of how many OTHER bugs, windlight or not, have been marked that way - perhaps not by you, but also by other people - just to get THEM out of the way too.

Not good.

Sin T. said...

Yes, thank you, Nicholaz, for putting this perspective on the table.
RL I've worked with Usability and User-Centred Design for many years and I have numerous first hands experience of users not giving a twinkers twat about any designer's or developer's intention.
For a developer today is already yesterday.
But we are not developers.
We are residents, here, today, and hopefully we will be that tomorrow as well.
But if today is broken, what trust will we have about tomorrow?
At the end of the day, the only reasonable question to the statement "It will be better later on" is "When?"

I am not in SL for the Windlight or what features may be launched next week.
I am there for the people I know today.

When will be Now, Oh, Philipus Rex?

Bob Bunderfeld said...

The issue with Philip is he doesn't understand what the "Here and NOW" is. He's got his head so far in the clouds, he's actually lost touch of what is become of the world of Second Life.

This saddens me a lot, considering I've sat with Philip on a number of occasions during BETA just chatting about the future and what he and WE were hoping Second Life would become. Back then Philip had a dream, one that he's already fulfilled and is now trying to expand upon. The problem is Philip, you forgot to check with US, your customers, your friends, your COMMUNITY, to see if WE agree that Second Life is ready to go forward.

After this last month of operation, Linden Lab should just offer every single paying customer a FULL REBATE, the connection issues have been horrendous, the ASSET Server issues still exists, the long lost TELEPORT bug is still in effect, and the SL Viewer crashes 1 out of every 2 times in use.

Instead of talking about the FUTURE of where we take Second Life, Philip should be talking about how we can FIX the current world and then CONNECT it to the Future.

BTW Nicholaz, you said in some earlier comment that you wouldn't ever go to work in a big company again, because you enjoy being your own boss. I wonder if Philip offered you FULL CONTROL of ALL programming employee's at Linden Lab would you work for him?

zanncanto said...

Your mention of Windlight brought to mind my initial reaction to the idea when it was first launched. I believe the power of SL is the ability to share creations and experiences with other users. Windlight is created to appeal to the individual experience. It is of little more interest than a passing curiosity to set my sky preferences unless I can share what I'm seeing with those near me, or set them for my land.

Dean said...

I once worked for a small startup with a CTO who was constantly at me to put my head in the here and now and not focus on "blue sky" projects unless time permitted. As a young, ambitious sysadmin, it was frustrating, but it was always the right call.

I also wonder if LL is plagued by the all-or-nothing disease. Can't solve the very heart of the problem? Don't even try. Or try, but feel all icky about it. Then, of course, become averse to dealing with immediate issues of usability and functionality altogether because they never solve "the real problem". No, LL, THEY DO. The real problem for users is that X doesn't work. They don't care that X doesn't work because of some fundamental issue that will be completely reworked in 6 months. You care, but that's your problem. You manage the blue-sky solution internally and FIX THE ACTUAL PROBLEM today. Will your blue-sky project even matter in 6 months when yet another blue-sky project has come about that will fix the new problems your first blue-sky project created? How many times must we repeat this process before it becomes about FIXING THINGS NOW?

Peter Stindberg said...

I'm still waiting for the "transparency bug" to get fixed - and that's almost a year since I first came accross it.

It was addressed about 6 months ago in a release, but the release did not solve it, and actually made it worse in some circumstances.

Anonymous said...

Great post, Nicholaz. I think most utopianists put things off to the "Bright Future". The Soviets were always doing that, there were a million jokes about it.

I wonder if you could comment on the issue of your proposal being closed -- you closed it, but under pressure -- to have double click clothes and have them be wearable. You then apparently separated it into components. I discussed this on my blog.

Prokofy

Anonymous said...

Also I am SO thrilled to hear you knock the Windlight madness. It really is insane. We do not need this. It is so optional. It definitely should not be added, as it severely lags everyone. We are far from being able to absorb it, and can't turn it off if we have it.

Prokofy

Samantha Chandler said...

Very well said Nicholaz.

I used to constantly bitch about how frustrating it was for Linden Labs to continually add new crap to Second Life rather than fix outstanding bugs. Instead of working teleports or reliable logins we get stupid crap like voice and windlight that ties up development for the better part of a year now.

I don't bitch anymore, I am tired and just sticking around until another game comes along.

Anonymous said...

Hmm...:) the comments reflect a few misunderstandings. It is important to understand that advancing the client does not necessarily mean freezing developments regarding windlight, havok4, voice etc... This view reflects a lack of understanding regarding how large scale software projects are undertaken. Please dont get me wrong, not blaming any body pushing these arguments. The are millions of other things not related to software engineering I know nothing about. The current problems will not be solved by having everybody stop what they are currently doing and focus on so-called primary functionalities.

Cristalle said...

Fixing the client **shouldn't** restrict development of voice or Windlight. The problem is the other way around: that voice or Windlight restricted fixing of the client. Knowing this, most people requested focusing on the current viewer, instead of voice or Windlight, which introduce another set of seemingly all-consuming bugs that destroy the here and now.

I like voice, but don't use it much. It seems stable (now). Windlight is a different bag of bugs.

Besides, at the town hall, the Lindens said that supposedly 70% of Linden programmer resources were devoted to stability. If that is true, it is not evident in the client, when barely any bugs get fixed. What we have seen since then is a massive decline in growth of the premium membership, an exodus of players, and a lot of people waiting for more stability before putting any (more) money into this game. Stability is not the only reason for those occurrences, but it is certainly a big part.

Nadine Nozaki said...

I have been talking about communication problems, this puts these info context and make is possible to understand how what mush be so bright people have so hard time communication with the rest of us the users, residents, customers. I have been scared that SL will break down and fail, because LL don't focus on the residents.

I can't wait for a good working windlight browser, however it most be patched to be usable, and what i hear the code don't compile. :^(

Sascha Vandyke said...

You are absolutely right. I can only call this ignorance. LL treats you as single user who reports issues as if it is your own fault and if you really have a proof they state: 'we need more info on that'and so on. If you report now in world a bug with the bug report tools, f.e. with BE-S, you will get an e-mail that says your version is too old, lol. I mean what about the HET-Grid and all that. It really looks like LL have put their focus on different things instead of their users. I could give many examples on bugs i reported where a patch was in jira available and it was not taken over in some month. I also have still a bug open that was just closed, because somebody said it works (has a different config) and then a linden saying: We need more info! (VWR-667). The consequence of all that is that i will not put any more bugs in jira now, it just doesn't help.

Nicholaz "The Mad Patcher" Beresford said...

Wow folks, I don't think I ever had a post with the comments pouring in so quickly and I appreciate each and every one.

I was pondering this topic a bit more yesterday and will probably share my thoughts in another post and will take Prokofy's comment and blog post to write a bit about the JIRA another day.

Meanwhile here are some quick answers/replies:

Gigs/all: Even if you take away 100 there are 60 left. I don't want to indulge in a body count here, the main point of the post is the difference in focus. You know from the office hours/meeting, that for the Lindens "imported to internal JIRA" means almost fixed and "fixed internally" to them means practically fixed. Residents and developers are mostly living in different time-spaces. Partly through their focus on visions and long term solutions, partly through their different reference points.


Bob: Interesting question, but I wouldn't take the job, even under these conditions. A job like this doesn't suit my personality and I don't believe in control over people. Organizations pretty quickly create a sort of momentum (through the type of people they hire, through the way things "have always been that way", through procedures and tools and habit) which over all is hard to change. LL going open source is one example for that.

Prokofy: I'll write more about my view of the JIRA another time, but with those issues, I didn't really expect them to accept these changes anyway (over the time I have developed an internal ProbabilityOfAcceptance-o-meter).

Prokofy/All: Regarding Windlight, I haven't seen it running on my own computer so far, just saw the pictures. I'll be amazed if it doesn't have negative impact on the FPS, but the issue and my thoughts about it are probably worth another long post when I first look at it.

All: Over all the main point of this issue is, that the Lindens are doing a good job *within their own frame of reference* (I really don't blame their developers). The point is that the whole company (led by Philip) is focusing on a greater future goal which takes it's toll in the present/now. Their own invest/return calculation (not necessarily in monetary terms but also in terms of Linden's values and the sense of achievement (individually and as an organization)) is based on different parameters, which naturally leads to frictions with the end users/residents.

Raul Crimson said...

Thanks, Nicholaz. Not only for the patches (as you said with your patches is always better), thanks also to enlight a bit the Linden Lab reactions. I really feel a bit lost with them, still don't know what they want to do, sometimes i think they are not sure either.
Personally i think the JIRA is just a place to "resolve internally", so maybe we should keep reopening issues.

joshuameadows.com said...

"How long does it take Linden Lab to replace a broken light socket? About six months. Because there are currently plans to completely renew the electricity in the house anyway, which you will understand takes some time, thank you for your patience."

This was the most beautiful thing I've ever read regarding LL before. Thank you.

Nicholaz "The Mad Patcher" Beresford said...

(for the record, I think that was Joshua Linden, the release manager for SecondLife (winner of my personal "Linden of The Year" (category "Best post on the official blog") award)).

Aliasi said...

Re: Windlight, it's not such an FPS hit, honest. For my system, it significantly improves FPS and looks; the old clouds (being giant particles) really slugged for me.

(Sky preferences saved at the sim level ARE coming - and Windlight happens to be very useful in establishing a 'look' for a sim.)

Doesn't mean that you aren't speaking the truth, mind you - while I realize LL's never going to be the homey little startup they were when I first joined, it'd be nice if they realize the big problem at this point is not features and options, but stopping leaks in the boat before it sinks. Nobody cares about gold-engraved shuffleboard if we're at the bottom of the ocean!

Celierra said...

I think you pretty much hit the nail on the head. :)

Also, the UI redesign is already semi-available if you want to play with it - see https://wiki.secondlife.com/wiki/Viewer_Visual_Update

Nadine Nozaki said...

Viewer look, all but any changes in the view look is available in that, it's just a skinning project. And maybe some wishes (not including the wish to make the communications box go away.)

The previe is also imho unusable until they can get the changes of the text colors. I can't read the chat or IM windows in that one and im part of the 75% of the users that don't think voice is a salvations of SL.

Anonymous said...

You hit it right on the spot. :)) thankyou very much for your focus on the now.
-Sylvia

Winter Ventura said...

in the mechwarrior online gaming community, there was(is) a group developing a "fan mod" which added a hundred new mechs, a dozen or more new weapons, and changes to eveything from weapon efficiencies, user interface, and even the cammoflage paintjobs on the mechs.

this project was not opensource. There was one person at the top, who had to code it all. He had managed to hack into the code, and had developed some programs to help him do this. But if this code were ever released, it would be open season on cheats, and the thriving (at the time) league play communities would have been decimated.

this group, depending solely on volunteer artists, volunteer modellers, volunteer testers, etc.. was unbeleivably slow. To the point of taking up to 2 years to release a single patch.

The phrase "Soon™" came into use by disgruntled fans and volunteers alike. "Soon™" seems to be precisely how Linden Labs operates.. precisely because their internal structure is "volunteer based". "When I get around to that, after he gets around to this, after those two guys get around to that, after this guy submits this and that guy decides if we should even bother"

There's jira entries dating back to the beginning of the Jira, that still don't merit the Linden's notice. There's jira entries on the chatterbox (my one big issue in the last 6 months) that the lindens have only said so much as "I'll look into this" or "let me show someone" and then the thing stops dead.. with TONS of votes.

frankly, LL in my opinion, is ripe for a Sony, Microsoft, or Google ass-fisting. (pardon my vulgarity).. Someone with money, and organization, and real programmers working as a team, with management and deadlines, experts contracted in, controlled testing and fast paced development. Focus groups, User Interface experts contracted, the works. Someone to literally blast by them and leave LL capsized and spinning on it's roof in the wake.

Someone to come in and make "the browser" and then "the server" and sell a million copies while LL is still mired in "identity verification" and "sexual ageplay" and "casinos". While they're still playing "net nanny" to their own private "compuserve", someone is going to make the internet.

The thing I'm starting to notice:

LL itself won't scale.