Musicians Collaboration Studio
How To => Using This Board => Topic started by: Gerk on April 06, 2020, 07:17:34 PM
-
Gi folks, as reported in the shout box some of us have been experiencing stalls when using the board. If you have this issue please report it in here around the time it happens if you don't mind. I know what is causing it (for any geeks that care there's something whacky happening with PHP cache) but not sure exactly how to fix it yet, so more evidence will help me track down the issues.
Thanks in advance, and hoping that the small tweak I made already resolved the issue ...
Gerk
-
Hey Buddy!
I was getting lockout after I chose to play a file from the studio A list. If I crazy clicked on refresh about 4 or 5 times, it would let go and let me get on my way.
Thanks!
NickT
-
Hi, I have visited a few times in the last handful of days and I though the problem was solved, but I just experienced it again about 5 minutes before this post. The browser was just spinning for 10's of seconds (until I killed it), failing to connect or start loading a page. At the time 32 users online (2 members, 31 guests). After I closed browser and reopened it I was able to load a page with the browser connecting after spinning for about 4-5 seconds. Now 5 minutes later pages are loading fast like usual.
George
-
same story, just had the first stall since Mark posted his small tweak fix. Took about 5 mins to get back in. Now business as usual again
-
Hmm ok I had made another couple of tweaks, has anyone seen this is the last week? This seems to be a tough one to pin down :(
-
Hmm ok I had made another couple of tweaks, has anyone seen this is the last week? This seems to be a tough one to pin down :(
On Saturday evening (Central Eur Time) the site was almost unusable Mark. Could get in for one action and then it would stall again
-
Hmm dang. Ok. The strange thing is that it only happens with this particular site, there are other sites on the same server that have no issues. I'll keep plugging at it.
-
I've made a couple more changes ... you might have to logout and back in again (changed up some stuff with the site cookies). I'm hoping this resolves the issues. If not I have other ideas but I don't like having to go there as it would involve disabling the html5 based play feature which I think it the current biggest culprit for these stalls ...
Please let me know if you see more site stalls.
-
Question ... when you experience the site stalls does it always relate to the mp3 player button? It seems to for me ... so trying to figure out if this is what's causing the issue.
-
Question ... when you experience the site stalls does it always relate to the mp3 player button? It seems to for me ... so trying to figure out if this is what's causing the issue.
Not always for me but started off like that a while ago, sometimes just can't log on at all. Other times when I'm in, I want to navigate to another forum or page and then it stalls. Sometimes I try to play an mp3 and it won't play. Lot's of different reasons
-
I think I figured out the issue ... the MP3 file preview using HTML5 source tag seems to be the root of the issue. The technical details are that because browsers try to call them all concurrently and download them it's causing a race condition with PHP's session file locking ... i.e. a whole bunch of things are trying to access the session file on the hard disk at the same time, so when these long timeouts happen it's because there's a bunch of things in a queue waiting to access the file and you can't get out of it until they are all done. On some pages because of the big list of files at the top that can mean that it was trying to download ALL of the full MP3 files before it allows you to continue. So if/when you hit one of these locked session files it would have to wait and download one at a time until they were all done.
I'm going to work on a new way to get those previews back up and running that won't trigger this file locking issue! It was not limited to just pages with MP3 previews (it could happen with anything with a bunch of attachments it can preview, like images) but it was not widespread until we moved to this new server with a more modern version of PHP.
It may take a bit of time for all your browser cache to clear, but once it does I suspect that the stalls will be gone. If anyone wants to be proactive about it you can look up how to clear your browser cache for this whole site. Best to remove cache and cookies (you will have to log in again).
-
So more small changes which hopefully resolve this file locking / session madness. Going to leave the MP3 player off for now until I know this has resolved the stalling issues. Since these last changes I haven't had a single stall .. so crossing my fingers that it was resolved and once I know that I will figure out a better way to have an audio player for previewing purposes...
-
So more small changes which hopefully resolve this file locking / session madness. Going to leave the MP3 player off for now until I know this has resolved the stalling issues. Since these last changes I haven't had a single stall .. so crossing my fingers that it was resolved and once I know that I will figure out a better way to have an audio player for previewing purposes...
will keep you posted :D thanks for the deep dive and hard work
-
Cool, let me know. The site is super zoomy for me now ... but time will tell.
-
Hi it stalled again for me about 10 minutes ago, and became accessible 5 minutes later. The only other registered use on at the time was snowdog.
around 20:30 UK time on Saturday 18th April
George
-
Ok thanks. I'm wondering if that was something that was still stuck in cache with the old MP3 player attached to it. I didn't see anything interesting in the logs at that point so that's my best guess... I haven't had a single freezeup since removing the player and clearing my cache.
-
Ok thanks. I'm wondering if that was something that was still stuck in cache with the old MP3 player attached to it. I didn't see anything interesting in the logs at that point so that's my best guess... I haven't had a single freezeup since removing the player and clearing my cache.
no problems at my end since the mp3 was disabled, was possibly once but I cleared everything in the cache recently too and haven't had a problem since
-
Ok that's good news. I've read a little bit more about this issue related to other stuff and I'm pretty sure that's what was going on here. The way this forum software builds its sessions is a bit old-school ... which is not the supported way of doing things anymore and it would be a significant undertaking to try and rewrite those bits. I'll see if I can figure out a different way of doing the MP3 previews. I have a pretty good idea but not sure if/how it will work. When I have a bit of time I'll try and hack it into place and see what happens!
-
Thanks mark. I just got a notice of the html 5 updates from codecanyon. I'll download and and throw them up in a folder for you to play with.
Cheers bud!
-
The thing I have to invent is a wrapper around it all so that it doesn't try to fire off a dozen (or more) sessions at once, that's what was breaking things. I can probably still use the previous player, just have to build some more javascript in front of it all so that they are only done on demand instead of on load!
-
Just slap a condom on it! ;)
-
I had another stall issue again just now :(
I'm honestly at a loss, the only thing I can think of is that the software we're using is super old and should not even be able to work with this modern version of PHP and the hack we are using is causing issues. It's nowhere near as widespread but still happens occasionally. Not sure I can do much about it at this point.
-
Hi
it's happening for me every day, and I only spend a few minutes on the site.
G
-
I had another stall issue again just now :(
I'm honestly at a loss, the only thing I can think of is that the software we're using is super old and should not even be able to work with this modern version of PHP and the hack we are using is causing issues. It's nowhere near as widespread but still happens occasionally. Not sure I can do much about it at this point.
i had a glitch posting yesterday. But it did post, just took a while. That was the first time in a long time.
-
I found another couple of small tweaks that I made. Kinda grasping at straws but might be related. Again please report back if you see any other stalls.
-
Stalled hard 05/05/2020
i was getting a site not secure message. But when the site can up, All was secure.
Site setting 3 cookies.
Nick
-
Stalled hard 05/05/2020
i was getting a site not secure message. But when the site can up, All was secure.
Site setting 3 cookies.
Nick
Ok thanks, that's good information to know. I think I'm on the right track then. It's definitely session/cookie related and the settings changes I made are related to cookies specifically (I restricted some things from being in cookies). This is likely a result of the changes I made and your browser doing some funky things trying to get them sorted out. For the insecure site thing it was probably because your browser auto-filled in an http instead of https and the stall happened on the redirect. I would suggest logging out and clearing all cookies if you want to be proactive, then the two cookies will be remade on the next login.
-
The post never finished back to the thread. Just hung,but the new post was there.
Interesting. I'm just giving info Mark. I figure the more you see, the better it can be used to flag something.
Thanks brother.
-
FYI. Site is screaming now.
-
mark,
Did you catch any strange logs at the time marked when it stalled?
-
No, all of these stalls are on client side, but initiated from the PHP session_start, so nothing shows up in logs. I haven't had any more issues but I haven't been on the site a ton. It does seem much faster now though, so crossing fingers that changing up those last couple of things helped.
-
it was stalling every day for me for months, kind of expected it each time, but surprisingly has worked every time since yesterday
-
Stalled several times today
-
For those that are seeing stalls, have you cleared you cache and cookies? I suspect that these stalls are due to things being stuck in cache and still trying to use old sessions or load older versions of pages (and still trying to load song previews).
-
And within a minute of me posting that it totally crapped out for me. I think I found one more thing that was happening and am working on a fix for it. It's definitely PHP session related. I found a session directory that had over 1 million files in it, so I think that might be part of the problem. For the fellow geeks in the crowd it looks like PHP session garbage collection was not working. For the non-geeks ... something busted, working on it.
-
Hmm I think it might be time to look at replacing this forum and software ...
-
But please do keep us in the loop if it stalls again. The hail mary fix seems to be still working for me, but unsure if it's just more whack-a-mole or if it's helped others too.
-
Because of the hack you had to use to get v1.1.21 to work with a newer version of PHP, perhaps you could un-hack it, and use PHP 5.3 which is what this version of SMF was built for, unless you're using another PHP script that has to run on a newer version of PHP.
If you're using headless Ubuntu server, there are ways to install and use different versions of PHP, I wrote a bash script that allows me to change Apache-PHP versions as well as the PHP CLI versions, from PHP versions 5.6 to 7.4 inclusive, using a menu driven script, so that if a forum or CMS script complains that it needs a newer version of PHP, then I just run the script, select a new version, then re-run the forum/CMS install script and check it again, and so on, until I get the right PHP version.
Currently I have installed on my Ubuntu server, PHP versions:-
5.6
7.0
7.1
7.2
7.3
7.4
And I can change to any one of them using the bash script.
If you want to have a look at the script I can post it on pastebin and post a link here.
-
Thanks but that's not going to work, PHP5 is beyond end of life, is unsupported and has some known MASSIVE security issues, which is the primary reason why we had to stop running on the old version. You don't need to copy and paste this info and continue to post it all over the place. The shoutbox AND this thread AND private messages are a bit overkill.
-
And got a dreaded stall again just a little while ago :(
There's probably not a lot more we can do here to try and fix this one, stand by as we figure out how to move forward with all of this.
-
And got a dreaded stall again just a little while ago :(
There's probably not a lot more we can do here to try and fix this one, stand by as we figure out how to move forward with all of this.
Playing whack-a-mole. One more tweak added. Shaking my head as to why the SMF programmers thought any of the heavy handed stuff they were doing would be a good thing ...
-
Has anyone had any more freezes in the last week? I haven't had any ... and probably just jinxed it ...
-
I get one every couple of minutes I am on the site. I cleared my browser cache when you said a few days ago.
-
Hmm ... I did totally jinx it as well because I got one literally two clicks after this post.
-
always stalls after successfully downloading an .mp3 or .wav etc..
-
I used the site pretty heavy the last two days. PM's, Downloading finished works, Uploading a mix, saying Hi to Noobs.
I wonder if it's a certain time of day. I didn't have a single stall.
Weird! ;D
-
what browser Nick?
-
I've been using Brave.
https://brave.com/ (https://brave.com/)
-
Me too, but it stalls a lot for me!
-
It's really hard to pin down in terms of reproducing the issues, that's the problem. I use the site and rarely get stalls but others have them very often. I know what is causing the stalls (PHP sessions and the heavy handed way that SMF forum software tries to manage them), but there's really not a fix for it aside from rewriting sections of the forum software. Due to changes in the way they handle some things updating to the 2.x version of the forum software would mean breaking the collaboration module -- they would need a full rewrite to work with that software, and if I'm going to spend time doing a full rewrite then I'm going to handle things differently so that we're not stuck like this again in the future. Have some good ideas for how to do that, it's just a matter of having the time to do it. Since covid hit I've been busier than ever.
I'm an IT/DevOps guy for my day job and with all the work from home I've been up to my eyeballs keeping up with supporting both the company I work for and my IT clients. I'm also an avid gardener so the next couple of weeks all my spare time is needed to get all my plants and such in ... the timing on those can't wait. But after that I plan on spending some quality time and getting a proper solution written and in place.
Stand by folks ... there will be good things in the near future :)
-
Hi Gerk, I am IT guy aswell, but these days a manager. Years back I was a developer, but not PHP, rather C++ and Java. If I cant help let me know.
I know others are IT people, maybe some have the right skills to help
George
-
Hi George ... thanks, but it's not so much the right skills, it's more that we've pushed this software many years past when it should have been retired. SMF 1.x (what we're running) shouldn't even be capable of running in modern versions of PHP. The only reason it's still hobbling along is that I found an unofficial hack that added in some stuff to give it backwards compatibility that I added to it.
If you haven't done so already try forcibly clearing your cookies and cache that's related to this site. Since I did that with the last round of fixes the freezes have been very rare for me.
I also tried to see if it triggers after downloading files and I don't have that issue either, so I would suggest to try that if you haven't already gbm and see if that helps on that front.
-
I cleared cookies and had improvements for maybe 1 or 2 days, and then it came back.
-
Hmm ok. How many cookies does the site give you? I think the specific error here is that there is some race conditions between the cookie and the PHP session and when those happen everything stops dead until the default timeout (which has to be higher than normal because we have to allow for large file uploads). I've seen multiple cookies sometimes as well which makes for a larger risk of this kind of deadlock.