openstack team mailing list archive

[Klaus Schürmann <klaus.schuermann@xxxxxxxxxxxxx>]

If the replication traffic is responsible for this raising network traffic for only 1.200.000 objects, how much traffic I can expect if I have 100.000.000 objects stored? 
The average size of my mailobjects are 120 kB. It's planned to use all 12 hard drive slots of my DELL R720xd with 4 TB drives. I have 5 storage nodes and 2 balanced proxy nodes. Will the replication traffic kill my system?

Here is a small part of my object-server-replicator log:
Jul  9 06:48:16 storage-node1 object-replicator Starting object replication pass.
Jul  9 06:49:13 storage-node1 object-replicator 9830/9830 (100.00%) partitions replicated in 57.56s (170.78/sec, 0s remaining)
Jul  9 06:49:13 storage-node1 object-replicator 1234597 suffixes checked - 0.00% hashed, 0.00% synced
Jul  9 06:49:13 storage-node1 object-replicator Partition times: max 0.0279s, min 0.0068s, med 0.0104s
Jul  9 06:49:13 storage-node1 object-replicator Object replication complete. (0.96 minutes)
Jul  9 06:49:43 storage-node1 object-replicator Starting object replication pass.
Jul  9 06:50:41 storage-node1 object-replicator 9830/9830 (100.00%) partitions replicated in 57.69s (170.39/sec, 0s remaining)
Jul  9 06:50:41 storage-node1 object-replicator 1234643 suffixes checked - 0.00% hashed, 0.00% synced
Jul  9 06:50:41 storage-node1 object-replicator Partition times: max 0.0365s, min 0.0068s, med 0.0104s
Jul  9 06:50:41 storage-node1 object-replicator Object replication complete. (0.96 minutes)
Jul  9 06:51:11 storage-node1 object-replicator Starting object replication pass.
Jul  9 06:52:09 storage-node1 object-replicator 9830/9830 (100.00%) partitions replicated in 58.31s (168.58/sec, 0s remaining)
Jul  9 06:52:09 storage-node1 object-replicator 1234688 suffixes checked - 0.00% hashed, 0.00% synced
Jul  9 06:52:09 storage-node1 object-replicator Partition times: max 0.0348s, min 0.0069s, med 0.0106s
Jul  9 06:52:09 storage-node1 object-replicator Object replication complete. (0.97 minutes)

-----Ursprüngliche Nachricht-----
Von: Pete Zaitcev [mailto:zaitcev@xxxxxxxxxx] 
Gesendet: Montag, 8. Juli 2013 19:22
An: Klaus Schürmann
Cc: openstack@xxxxxxxxxxxxxxxxxxx
Betreff: Re: [Openstack] [SWIFT] raising network traffic o the storage node

On Mon, 8 Jul 2013 13:50:38 +0000
Klaus Schürmann <klaus.schuermann@xxxxxxxxxxxxx> wrote:

> I use a swift storage as a mail-store. Now I have about  1.000.000 objects stored in the cluster.

> Traffic Storagenode: http://www.schuermann.net/temp/storagenode.png
> Traffic Proxyserver: http://www.schuermann.net/temp/proxyserver.png
> 
> Can someone explain such behavior?

At a guess, raising number of objects makes number of partitions to
increase, and that increases the replication traffic, specifically
the number of sent MD5s for partitions.

It would be interesting to corellate the number of objects and
number of non-empty and empty partitions with the amounts of traffic.
If the increasing transfer of hashes is the reason, you could also
verify by graphing the traffic to port 873 separately. Swift never
replicates object bodies through its own HTTP interface, so this
splits control traffic from data traffic for you. Data traffic
should be driven by customer and node failures, not consistency
checking.

Be prepared to split up storage nodes, however. Even if there's no
bug with replication, its aggregate traffic increases with the
increases in object counts.

-- Pete