How do they do it?

5th June 2006, 12:01

Last night i was wondering something:

On a lan, when on Gbit, you can transfer data at 40-50mb/sec to 5-10 other users. And then your system is under heavy load.

But what with for example Eweka? Or Newshosting / Giganews? Just for easy calcution, let's say they got 10 000 customers connected on average.

Each one on broadband, again for easy calculation let's say 500kb/sec.

Bandwith isn't a problem, but what with the userloads and hddspace? Again for easy calcution, usenet is 10TB big.

With one 'heavy' desktop server you got 10-20 hdd's * 250GB = max 5TB, no redundancy, and a max of 50-100 users before your hdd's are overloaded. This will happen MUCH sooner if everyone is accessing something different on the same hdd's.

Just for the content you'll need 2 of those, and then 2 backups, or a raid 5-10 array. And that for only 50-100 users, do that 100x for the 1000 customers and you got A LOT of server.

And that's very conservative, because there are much more than 10 000 users, and for 80 days retention you'll need hundreds of TB.

So what kind of systems do they use? Ofcourse there a big serverclusters with thousands of cpu's, tb's of ram, but i've never seen or heard from a system where you can, for example, plugin 200 hdd's in raid 5.

jmke · 5th June 2006, 13:06

we're talking most likely here about Fibre attached external storage devices, short name is SAN.

all the big companies have them like HP and Dell; take this one for example:

Dell | EMC Fibre Channel CX700

Storage Capacity
Scales to 117TB

Supported Drive Types
15,000 RPM Fibre Channel
10,000 RPM Fibre Channel
7,200 RPM SATA

Drive Quantity
Up to 240 drives

Number of Hosts
Up to 256 servers

and now the kicker:

DATA bandwidth: 1300MB/s (or 1331200 Kb/s)

and they can easily place multiple ones like these

5th June 2006, 16:20

Aha those are the things i had in mind, but i had no idea they existed.

De item die u heeft geselecteerd is niet online te bestellen. Gelieve uw vertegenwoordiger te bellen om te bestellen.

FU dell!

jmke · 5th June 2006, 16:25

for 100TB you're looking at close to €30.000 I think

5th June 2006, 12:01	#1
teki Posts: n/a	How do they do it? Last night i was wondering something: On a lan, when on Gbit, you can transfer data at 40-50mb/sec to 5-10 other users. And then your system is under heavy load. But what with for example Eweka? Or Newshosting / Giganews? Just for easy calcution, let's say they got 10 000 customers connected on average. Each one on broadband, again for easy calculation let's say 500kb/sec. Bandwith isn't a problem, but what with the userloads and hddspace? Again for easy calcution, usenet is 10TB big. With one 'heavy' desktop server you got 10-20 hdd's * 250GB = max 5TB, no redundancy, and a max of 50-100 users before your hdd's are overloaded. This will happen MUCH sooner if everyone is accessing something different on the same hdd's. Just for the content you'll need 2 of those, and then 2 backups, or a raid 5-10 array. And that for only 50-100 users, do that 100x for the 1000 customers and you got A LOT of server. And that's very conservative, because there are much more than 10 000 users, and for 80 days retention you'll need hundreds of TB. So what kind of systems do they use? Ofcourse there a big serverclusters with thousands of cpu's, tb's of ram, but i've never seen or heard from a system where you can, for example, plugin 200 hdd's in raid 5.

5th June 2006, 13:06	#2
jmke Madshrimp Join Date: May 2002 Location: 7090/Belgium Posts: 79,021	we're talking most likely here about Fibre attached external storage devices, short name is SAN. all the big companies have them like HP and Dell; take this one for example: Dell \| EMC Fibre Channel CX700 Storage Capacity Scales to 117TB Supported Drive Types 15,000 RPM Fibre Channel 10,000 RPM Fibre Channel 7,200 RPM SATA Drive Quantity Up to 240 drives Number of Hosts Up to 256 servers and now the kicker: DATA bandwidth: 1300MB/s (or 1331200 Kb/s) and they can easily place multiple ones like these __________________

5th June 2006, 16:25	#4
jmke Madshrimp Join Date: May 2002 Location: 7090/Belgium Posts: 79,021	for 100TB you're looking at close to €30.000 I think __________________

5th June 2006, 16:20	#3
teki Posts: n/a	Aha those are the things i had in mind, but i had no idea they existed. De item die u heeft geselecteerd is niet online te bestellen. Gelieve uw vertegenwoordiger te bellen om te bestellen. FU dell!