New servers and a new IT team...

[ReefGuy]

SQL Server, or Oracle provides much better tools to handle these problems than mySQL. At least when I last worked with it, mySQL had a much more limited toolset than transact sql.

MySQL works fine at the level that scubaboard is at.


Since scubaboard is locking up under load on a roughly daily basis on a few servers, it is almost certainly race conditions in the vBulletin code where it isn't handling concurrency correctly. Throwing larger servers at the problem will help to make more idle cycles available will help to reduce but will not eliminate the race conditons. The code involved with the race conditions will also tend to be the worst code in the codebase since its the stuff dealing with concurrency, locking and generally resources like databases or networking which touch the underlying hardware. .

 

[mikerault]

So you're an Oracle DBA, huh?

It's a merge, sort, split and write for web logs, which were running between 50 and 100GB/day.

The C app opens all the input files and output files simultaneously and uses a hash map to find the correct output file for each record in each input file. The inner loop consists of mostly nothing but raw file I/O and a few pointers to data structures.

Because the import and export are simultaneous for all files, and there is no seperate import or index and virtually no search overhead, it's much faster than Oracle, which requires that the data be imported and indexed before running a query for each output file, then writing out the results to the filesystem.

There's pretty much no way a general purpose database even has a chance of beating an app hand-coded for a specific application.

Terry

Yep, right up to the point it crashes and needs recovery...

 

[lamont]

SQL Server, or Oracle provides much better tools to handle these problems than mySQL. At least when I last worked with it, mySQL had a much more limited toolset than transact sql.

You get a better toolset with SQL Server or Oracle at the cost of a closed-source codebase (and at a lot of $$$ cost). Its a tradeoff. There's still nothing indicating that MySQL (or postgres) are an inappropriate platform for scubaboard, I know for a fact that it works fine as a platform for simple applications under moderate load. The issue is going to be how vBulletin uses MySQL...

 

[Web Monkey]

Yep, right up to the point it crashes and needs recovery...

The source files are never written to, so there's no need for recovery.

If the process dies, just restart the app.

Bulletproof is good.

Terry

 

[mike_s]

It's amazing how FreeBSD and Linux have taken over the corporate web hosting world.

I mean how often do you see an IRIX or Solaris (or even HP-UX) web server out there anymore?

(I took my "IRIX Forever" sign off the cube a long time ago... )

 

[TheRedHead]

Webcom ran Solaris and Sybase before being gobbled up by Network Solutions. It was a horrible platform for web scripters.

 

[RonFrank]

So you're an Oracle DBA, huh?

It's a merge, sort, split and write for web logs, which were running between 50 and 100GB/day.

The C app opens all the input files and output files simultaneously and uses a hash map to find the correct output file for each record in each input file. The inner loop consists of mostly nothing but raw file I/O and a few pointers to data structures.

Because the import and export are simultaneous for all files, and there is no seperate import or index and virtually no search overhead, it's much faster than Oracle, which requires that the data be imported and indexed before running a query for each output file, then writing out the results to the filesystem.

There's pretty much no way a general purpose database even has a chance of beating an app hand-coded for a specific application.

Terry

The whole things sounds rather offbase. Why would anyone need/want/have 50~100G of WebLogs? Why would anyone attempt to store such in a DB? Why would one want to split those logs into multiple flat files? Who is going to look at 50G of TEXT data a day?

50G a day is no strain on Oracle, or a solid UNIX server, and there should be no need of two FULL time DBA's to handle that workload. This assumes adequate design, data scrubbing and archiving, and capacity planning.

The only time we look at WebLogs is when we are having issues. A historical copy of them would not be mandidated by any regulatory agency, and why would anyone want to parse them into different files, and keep them around?

If these things were getting generated by the application, the simple solution is stop doing that! If they are useful for debugging, then put in a verbose option, and leave it off unless you need it.

In any event, I seem to see the problem here. It has little to do with technology, and more to do with design, and appropriate use of technology. We use Oracle. We use a LOT of things (including Linux and MySQL). Ironically people have all sorts of beliefs on what is best, when in fact they are just tools, and one chooses what is best for the needs of the applications, and users.

 

[FastAttack]

Clustered doesn't necessarily mean scalable.

mySQL is a technically inferior platform that just happens to come with a really sweet price tag. There are other options now for more robust dataengines that aren't prohibitively expensive. MSSQLServer and even Oracle now have an entry-level offering for just such environments as this. (Driven to it, no doubt by the free nature of mySQL, etc.) What you get for your money that mySQL lacks is the *-abilities (recoverability, scalability, dependability, etc.)

I understand WHY websites are deployed on mySQL, and I'm fine with it as long as I'm not the one who has to recover corrupted tables, etc. (Been there, done that, not doing it again. )

Peace,

John

Hmm no its not technically inferior, as a standalone out of the box, maybe I'll agree with you.. .

dependability? hmm sorry to say that some of the longest uptime DB servers are mysql servers.. the main diffrence is that these db servers are running mysql with added private development with added toolsets created by the companies running it Most of the credit card processing companies use MySQL variant on a private developed *nx environment with their own development tool sets to run it, how do I know that? I am in that industry.. Visa and Mastecard have both Oracle and mysql transaction based DB machines).

recoverability?
hmm actually again what toolsets are we talking about.. open source or private toolsets for data recovery?.. Wanna talk nightmares.. MSSQL servers tend to be a pain in the *** to recover from a fatal error on a DB. You can't bring it back no matter what you do ( this is why you run MSSQl usually in a cluster setup with redundancy and no automatic re-write on the failover until the data is confirmed not to have errors)
although the newer versions of MSSQL are getting better at this

scalability?
Hmm of course it is scalable.. if you have the money you can invest on it and create the necessary scalability. it seems you are basing facts on just the standard off the web open source engine , and even as an open source. you can scale it to any OS you are running

Again I am not saying you are not right or I am right.. but the statements you made are pretty debatable.

 

[Web Monkey]

problem which is software-generated, but finding Windows people who can drill down to that level is extremely rare -- once you hit the point where the GUI or registry doesn't have a button you can hit and microsoft doesn't have a patch, you are stuck.

You mean like disassembling the MBP COBOL Loader, fixing the memory leak then reassembling it for $400.

I wish I had a clue how dumb I was back then. It should have been $40,000 not $400. That probably worked out to about $1 per location it was running at.

Oh well.

Terry

 

[mike_s]

Webcom ran Solaris and Sybase before being gobbled up by Network Solutions. It was a horrible platform for web scripters.

The biggest problem with platforms like that (or the ones I mentioned) is the support costs... yikes!