Showing posts with label grid status. Show all posts
Showing posts with label grid status. Show all posts

Grid Issues: Old IMs Reappearing in Chat

We have had reports of old IMs, some as old as 2 months, reappearing in chat as if they were just resent. We want to assure residents we believe this is a technical problem and not associated with anything intentional or malicious. We are investigating the cause and if you wish to send in a support ticket with your experience we will pass that information on to the investigative team.

Posted on Wednesday, December 9th, 2009 at 6:11 AM on Grid Status Report.

This bit of news was already known to some, judging by the chatter in some groups, "You mean they just noticed that?" One lady claimed to have gotten someone's reply to a question a little late, five months late.

Glitches and Crashes a personal view

Since Christmas older residents have had a strange sense of da ja vu. Crashes every 5 minutes, the "black body" as your avatars textures refuse to load - a return, in fact to the "bad old days". Now for us old timers this is just a minor irritation, in fact many of us suspected that it over enthusiastic staff parties at first, combined with the usual in rush of new residents.
I think most of us expected things to settle down and get back to usual once the holidays were over, but, in fact the strangeness continued. In fact in the last few weeks things have started getting really strange.
The first clue was a friend who burst onto my im panel in a flurry of swear words - some of which even I didn't know (you have to admit, Second Life is educational). After she calmed down I managed to ask what was wrong, it seems she had just settled down to some quality time with her significant other, their first quality time in a while, everything was going well when she crashed - annoying but it happens and can be laughed off, but on relogging she gets the "region has begun log out process your account wont be available for 5 minutes", annoying but as this seems to be a default glitch she clicked ok and tried again - success! Except she had been logged in, naked (well more or less naked, if you know what I mean - she had been on a spending spree in Insolence), to an info hub FULL of people... quickly she got her friend to teleport her home, ahh oh well, a funny story, except 5 minutes later the scene was repeated only she got logged in to a different region. In the space of 45 minutes she had crashed, and been logged into to anywhere except where she wanted to be SEVEN times, it somewhat killed the romantic mood. I laughed, I admit, and put it down to just one (or two) peoples misfortune, but I started paying more attention to the chatter in the bars, seems my friend wasn't alone, the crashes were happening to everyone, more and more of the time - my own personal record was twelve crashes in an hour. Still switch to another viewer and things improved.
Unfortunately that wasn't the end of the surprises LL had in store, used to these old style glitches and ready for something new? Linden Labs is proud to present the newest glitch...phantom land! A friend owns an island, a lovely island - and was most disturbed to log in and find herself sinking, at a quite genteel rate, into the ground. After a quick look around she logged out thinking it must be one of those, "clear cache, log back in" glitches...but no, after a relog she was still up to her neck in her beach. A quick phone call to LL and many apologies later it is explained that the servers in AZ are having issues. (Really, servers, which need to be kept cool, being housed in a desert and they are having problems - fancy that). Linden Labs, did manage to relocate the phantom island so the actual island was there and the phantom was just off the coast.
And now I come to the final and most distressing glitch, not a new glitch by any means but refined to a level rarely seen before. Missing inventory.
Usually a few items go astray and you clear cache, log back in and bingo - there it is, but not in this new more fiendish version. Suddenly the hair you are wearing isn't in your inventory, it doesn't exist, it is still there, everyone can still see it, but you can't take it off or replace it, or detach it because it isn't in your inventory (only funny if the neon pink pigtail demo is being worn by someone else when it happens).
A friend of mine has the interesting problem of three quarters of his inventory vanished - but only on his home computer, on his works computer everything is still there, a complete reinstall of SL didn't help and the official response to LL help line was "That's odd, don't really know what to recommend", honest, if nothing else.
Another friend had all the outfits for a photographic shoot vanish - for a week, in a flat spin she asked me to take charge of the men's outfit, on the day of the shoot, I had given out 2 of the outfits when I crashed. On my return I was missing almost four thousand of my inventory items including the last suit - relogging, clearing cache - nothing worked, the suit was GONE. A frantic call to the designer and a replacement was found. Twenty four hours later, the suit (and the other four thousand items)were back. A curious twist to this is something I discovered, after my brush with missing inventory I decided that really, twenty five thousand items is a bit much (don't look at me like that, shopping is my hobby and I was a lucky chair addict), so I pruned, I spent a morning doing nothing but deleting things. I emptied my trash, I cleared my cache and I logged out, pleased that now I was down to under twenty thousand items. Imagine my dismay when I logged back in later that day to find all my inventory there exactly as it was before my mega clean out. The moral might be, don't organise your inventory, leave to run wild and let it surprise you.
What is the cause of the sudden instability of our beloved SecondLife? I can't answer, obviously LL are aware of the problems and are working on the fixes. Meanwhile, my best advice is only make sure you are wearing something at all times, and preferably something that preserves your modesty and you like enough to wear for a while if the worst happens.

Weekend Grid Outages

Although the second half of 2008 showed a big reduction in usage hours lost to outages (we reduced outage hours by over 50%), stability challenges have increased over the past month on the Grid. This weekend was especially painful, and the first time since joining Linden Lab that I’ve experienced a full mySQL crash (this occurred just after 4pm PT on Sunday).

When the central database crashes, it takes approximately 1 hour to rebuild tables and indexes before accepting queries and becoming fully functional. This was the main reason for us to employ the painful triage process of temporarily blocking logins, while the database is in an overload state. These 5-10 minute “blocking periods” are substantially less Resident impacting than a full database crash, and a 50-60 minute restart cycle. However, neither is acceptable and I wanted to continue updating our efforts to stabilize the infrastructure.

You can read the rest of what Frank Ambrose (FJ Linden) has to say HERE

But I personally would say a great deal more is wrong than shows on the surface? How many of you can get to the support page from the secondlife website? or even from individual articles - I had to find a backdoor through the wikipedia to be able to submit a support ticket.
How many have had their credit card suddenly rejected after a long time using the same one?
Some have shown signs of these problems and these are on top of the actual inworld issues which are plaguing us all
I for one wonder what exactly is going on at Linden Lab just now as they seems determined to drive its most staunch Residents out.

Post Christmas Grid blues

Each year, after the Christmas gifts have been unwrapped and all those people who treated themselves or family to new computers/Internet access, Secondlife sees a surge of new signups and more people using it than ever - the problem with that?
Quite simply the grid cannot handle it!
Most of us who spend regular time inworld are aware of the issues and mostly what is behind it - basically once the grid reaches 75k+ online everything goes down the toilet - teleports, transactions, rezzing, building even dressing becomes problematic.

Lets have a look at the last few days:

[RESOLVED] Logins and In-world Issues
Posted by Status Desk on January 6th, 2009 at 10:45 am PST

[Resolved 10:45am PST] - The database has recovered and all issues have been resolved.

[10:30am PST] - We are currently experiencing a problem with the database. There may be difficulties with logging in, rezzing objects, completing transactions, and searching. We are aware of this issue and our ops team are working to resolve it as swiftly as possible. In the mean time, please avoid making important transactions or rezzing no-copy objects.

[RESOLVED] Inworld Issues/Logins Disabled
Posted by Status Desk on January 6th, 2009 at 04:10 pm PST

[RESOLVED] Inworld Issues
Posted by Status Desk on January 7th, 2009 at 02:58 pm PST

[RESOLVED] Inworld Issues
Posted by Status Desk on January 8th, 2009 at 01:34 pm PST

[RESOLVED] Logins Disabled
Posted by Status Desk on January 10th, 2009 at 01:54 pm PST

[RESOLVED] Logins Disabled and Inworld Issues
Posted by Status Desk on January 10th, 2009 at 04:53 pm PST

[RESOVLED] Logins disabled
Posted by Status Desk on January 11th, 2009 at 11:20 am PST

[RESOLVED] Logins disabled
Posted by Status Desk on January 13th, 2009 at 03:39 pm PST


Each of these represents times when Logins have been restricted, a black week for any Secondlife business.
Some have called for allowing only premium accounts to log in at these times and yet others slam this notion as many who would like to go premium cannot do so, therefore making the idea unjust.
Then there is the question of bot activity, if as Linden Lab numbers suggest, 10% of those signed in are bots then surely shutting these out would allow real players more of a chance to log in?
But how do we or Linden Lab know which accounts are bots?
Age old question and one many have tried to answer - but to be honest our answers do not matter Linden Lab will do what Linden Lab chooses as always.
I am not technically minded but even I can see the grid is struggling - Linden Lab says 10% bots - most believe that is a very conservative estimate it is more likely to be around 25 - 30%. Now how about if the bots were registered? Then when the Grid is struggling bots are logged out? Wouldn't that make more sense than locking out paying Residents? Even those who run bots would rather have paying Residents inworld than their bots surely?
Hopefully with all the grid issues Linden Lab will finally make a move on the traffic scam that causes most to use bots - once the reason for them is gone it will make the need for bots redundant. But for now Linden Lab have chosen to chase their tails and deal with issues as they arise rather than dealing with the root cause (as always).

Dana Vanmoer

[Resolved] Land Store, Logins and In-World disruptions. (RESOLVED - honest)

Amid many celebrations and anniversaries this weekend is the Grid status page has left many thinking the problems teleporting, transaction failures and basic lag issues are only a problem for them;

[Resolved 12:00pm PST] - The Land Store has been re-opened and is functioning as normal once again.
[Update 11:35am PST] - All services have been resolved and are working as normal again, with the exception of the Land Store which will remain closed while we test functionality. We will update this blog when the Land Store has been re-opened.
[9:28am PST] We are looking into issues affecting transactions, logins, account creation and possible extensive related disruptions of services. Please refrain from any transactions as possible while we work to resolve this issue. During this time the Land Store has been closed to avoid further complications.


Guess what guys it is not RESOLVED I have spoken to many people since this was published as a resolved issue and believe me it UNRESOLVED but of course we have no way of letting Linden lab know this since we still cannot comment on the grid status page
Lets hope it actually becomes resolved before our own events begin tomorrow!
Dana Vanmoer

Rolling Restart to deploy 1.24.6, Tue-Thu Sep. 23-25

If this is what you get when you log in don't panic:
There will be a rolling restart this week to deploy server version 1.24.6 to Second Life. Release notes for what is changed may be found on this page on the Second Life Wiki. There are no new features in this release, only bug fixes. The version that will be deployed to Second Life is currently on the Preview Grid on regions running the Second Life Beta Server channel.
The rolling restart will follow this schedule:
Wednesday, 8:30-10PM: a pilot group of ~3000 regions will receive 1.24.6.
Thursday, 5-9AM: half of Second Life will receive 1.24.6.
Friday, 5-9AM: the rest of Second Life will receive 1.24.6
Each region will go down for about 5-10 minutes once during the rolling restart. If your region stays down for more than 20 minutes, please contact support.
Discussion of this rolling restart can be found in
this thread in the Second Life forums.

The rolling restart is to fix 2 serious bugs in the last deployment on the forums you will find some interesting discussion and its nice to see Prospero Linden responding reasonably to most of the sensible points on there.

UPDATE: "the server deploy has been postponed a day, and will start either Thursday evening or Friday morning."

Let us hope that the updates go smoothly - I know many have been experiencing an increase in crashes since the mono rollout - fingers crossed everyone.
Dana

Looks like its gonna be a bumpy weekend for Residents!

Rolling Restart to deploy server version 1.24.5, Wed-Fri, Sep 10-12
We will be deploying server version 1.24.5 to Second Life this week. This version includes no new major features, but fixes bugs with the current 1.24.4 server that is on Second Life. More details of what’s been changed can be found on the Second Life Wiki’s release notes page for 1.24.
This rolling restart will follow our usual three-stage deploy:
Wed, 09/10, 9pm : A pilot group of ~3000 regions will be restarted.
Thu, 09/11, 5-9am : Half of Second Life will be restarted.
Fri, 09/12, 5-9am : The remainder of Second Life will be restarted.
As with all rolling restarts, each region will receive warnings starting 5 minutes before they are restarted. No region should stay down more than 10 or 20 minutes; if your region stays down longer than 20 minutes, please contact support.
Further discussion of the 1.24.5 server deploy can be found in this thread in the Second Life forums.

Server Crash Affecting Logins
[6:59 AM Pacific] One of our main infrastructure servers has suffered a crash. This will result in various in world problems as well as failed logins. Our technicians are working on damage control and we hope to have things back to normal and logins fully restored shortly.


Prospero Linden
The problems we had before (an hour or two ago) were not due to the rolling restart. A technician in our datacenter hooked up a cable, and somehow a network loop was created. We figured out what was going on pretty quickly, and asked them to disconnect the cable. In the mean time, however, there was a fair amount of chaos. We're not sure exactly how the cable was connected incorrectly...............
In case you didn't see it on the status blog -- we're postponing this morning's rolling restart, as we have one more bug fix that's going to go in. Assuming all goes well, we'll redo the pilot roll later this morning, and then will do a Thursday Night / Friday Morning half-and-half roll of the entire grid.

Warning GRID ISSUES

There is nothing on the blog (as usual) Nothing on grid status page either
but I have been having problems for a few days now with LAG levels being high, failures on teleport and rezzing objects.
Today attempts to rezz almost invariable fail - I would not attempt to buy anything right now either - teleport sometimes works and sometimes doesn't.
Right now I am stuck on the log in screen after 20 minutes and 3 attempts to log back in after trying everything else.
My guess is that maybe it has some connection to the 1.24 Server deploy but that is just a guess - timing seems to much to be a coincidence - come on guys I guess you got it wrong again!
Time 06.30am SLT
I will update you if we get to hear anything.
Dana

UPDATE:
Live chat help -
Aurelie: Hello dana_Vanmoer, what can I help you with?
dana_Vanmoer: I can't log in
dana_Vanmoer: just get stuck with the bar about a 1/4 way up
dana_Vanmoer: I have tried several times in the last 20 minutes
dana_Vanmoer: hello?
Aurelie: it is a known issue
dana_Vanmoer: so why isnt it on the grid status or blog?
dana_Vanmoer: and is there a solution or time frame?
Aurelie: they are trying to fix it
Aurelie: and they are going to put a note on the blog
dana_Vanmoer: so we dont need to know because they are trying to fix it?
Aurelie: not yet
dana_Vanmoer: so i just wait for the 'never there' blog updates?
dana_Vanmoer: a warning on the log in page may be a better alternative instead of people wasting their time
Aurelie: is there anything else i can help you with?
dana_Vanmoer: yes give me some time frame?
dana_Vanmoer: or you suggest i just keep trying?
Aurelie: keep trying
Aurelie: some accounts now manage to log in
dana_Vanmoer: ok thank you for your help
dana_Vanmoer: not much point logging in if nothing works

UPDATE:
[RESOLVED 6:44AM PDT] The problem has been fixed. Logins and teleports should return to normal once the initial rush to log back in subsides.
We are having a problem with a VPN that we are currently investigating. This will have an effect on teleporting as well as blocking some residents from being able to log in. We will have updates on the situation as soon as possible.
(we shall see)