New servers and a new IT team...

[Web Monkey]

Forgive me if I fail to believe...unless the Oracle app was totally screwed up from the start and way over spec'ed there is no way a single processor small C app is going to replace 16 processors unless they were terribly old and slow. There is obviously something left out of the story...

Mike

So you're an Oracle DBA, huh?

It's a merge, sort, split and write for web logs, which were running between 50 and 100GB/day.

The C app opens all the input files and output files simultaneously and uses a hash map to find the correct output file for each record in each input file. The inner loop consists of mostly nothing but raw file I/O and a few pointers to data structures.

Because the import and export are simultaneous for all files, and there is no seperate import or index and virtually no search overhead, it's much faster than Oracle, which requires that the data be imported and indexed before running a query for each output file, then writing out the results to the filesystem.

There's pretty much no way a general purpose database even has a chance of beating an app hand-coded for a specific application.

Terry

 

[DandyDon]

.

​

.

 

[letterboy]

I know right don I know dude. . .

 

[RonFrank]

check out www.humanclock.com . They run their servers on an old TRS-80 model 100 and an old Atari (from the 70's or early 80's). No I'm not kidding.

All kinds of ways around that....

Based on a quick look at the site, I serously doubt they run ANYTHING on an old TRS-80.

"Humanclock.com is a website handcrafted using the finest UNIX text editors.
This website also exists due to the efforts created by the fine authors of the following "free" software packages which were used to create (and maintain) this website:
PHP, Perl, ImageMagick, mySQL, GNU Project, Redhat Linux, and FreeBSD."

You SERIOUSLY expect people to believe that they would even bother attempting to run a server on such hardware? It would be more of a headache keep up the old junk vs. buying something 1000 times faster, more upgradable, and easier to maintain (hardware and software) for a few hundred bucks.

Where did you get this information?

 

[AstroDad]

While I haven't seen this type of extreme, it's not entirely out of the realm of possibility. I've seen some really poorly written Oracle apps... real bad. Hard to believe someone got paid for it...

Forgive me if I fail to believe...unless the Oracle app was totally screwed up from the start and way over spec'ed there is no way a single processor small C app is going to replace 16 processors unless they were terribly old and slow. There is obviously something left out of the story...

Mike

 

[Sideband]

I don't mean to complain, but I still find that every single day there are one or two periods of an hour or so that I can't connect. I get a connection timed out error. I just went through one and am now able to connect.

Joe

 

[mike_s]

Based on a quick look at the site, I serously doubt they run ANYTHING on an old TRS-80.

"Humanclock.com is a website handcrafted using the finest UNIX text editors.
This website also exists due to the efforts created by the fine authors of the following "free" software packages which were used to create (and maintain) this website:
PHP, Perl, ImageMagick, mySQL, GNU Project, Redhat Linux, and FreeBSD."

You SERIOUSLY expect people to believe that they would even bother attempting to run a server on such hardware? It would be more of a headache keep up the old junk vs. buying something 1000 times faster, more upgradable, and easier to maintain (hardware and software) for a few hundred bucks.

Where did you get this information?

I first learned about that site from an article I read about how they were doing it.... I was just pointing it out and what it ran on for humor purposes.

But I gotta agree with you 100% that attempting to run that config on that hardware would be more of a headache verses running it on something more modern and efficient.

 

[lamont]

Give me raid 10 over 5 anyday, as long as you can afford the overhead.

But that still leaves the issue of the database. To my knowledge (please correct me if I'm wrong), mySql is not scalable. It is nice, however, if you're not running win boxes, but if you are, sql2003 run in a cluster is a far better route at this level.

MySQL works fine at the level that scubaboard is at.

Linux does not work particularly fine once you start to saturate the CPU on your machines along with having simultaneous high I/O loads, although usually it still takes a couple months or so of continous running to start to see the lockups (and in 2.4.x we're talking fairly nasty stuff like deadlocks in the journal block driver under the filesystem code -- I don't have the level of experience with 2.6.x yet to tell you what the problems are there in detail). Once you get to a few hundred Linux webservers you should start to see them going down every few days, at the level of 2,500 Linux webservers you'll see a few machines a day crash. I don't think that Microsoft does any better with windows at this scale, though (did MSN finally get off of Apache/FreeBSD and onto Win/IIS?).

Since scubaboard is locking up under load on a roughly daily basis on a few servers, it is almost certainly race conditions in the vBulletin code where it isn't handling concurrency correctly. Throwing larger servers at the problem will help to make more idle cycles available will help to reduce but will not eliminate the race conditons. The code involved with the race conditions will also tend to be the worst code in the codebase since its the stuff dealing with concurrency, locking and generally resources like databases or networking which touch the underlying hardware. Windows isn't a solution to the general problem. You need a either a better codebase than vBulletin or people who understand the vBulletin internals better and can fix the problem (and typically most database engineers or software developers do not understand how to debug problems which are close to the metal -- they spend most of their lives writing schemas and business objects, which is not the skillset you need). The nice thing about Linux and open source software is that if you have people who understand the internals you can fix nearly any problem which is software-generated, but finding Windows people who can drill down to that level is extremely rare -- once you hit the point where the GUI or registry doesn't have a button you can hit and microsoft doesn't have a patch, you are stuck.

 

[lamont]

Based on a quick look at the site, I serously doubt they run ANYTHING on an old TRS-80.

"Humanclock.com is a website handcrafted using the finest UNIX text editors.
This website also exists due to the efforts created by the fine authors of the following "free" software packages which were used to create (and maintain) this website:
PHP, Perl, ImageMagick, mySQL, GNU Project, Redhat Linux, and FreeBSD."

You SERIOUSLY expect people to believe that they would even bother attempting to run a server on such hardware? It would be more of a headache keep up the old junk vs. buying something 1000 times faster, more upgradable, and easier to maintain (hardware and software) for a few hundred bucks.

Where did you get this information?

If they said they ran the website on NetBSD on top of a TRS-80 or Atari I would tend to believe them... The NetBSD people are porting freaks...

 

[TheRedHead]

I believe the former servers were running FreeBSD.