openstack team mailing list archive

[Shashank Sahni <shredder12@xxxxxxxxx>]

Hi everyone,

I'm running a small test setup of swift with one proxy+storage and one
storage instance.

The setup seems to be running well for small objects. I'm able to create
and upload them successfully using swift cli tool, but for large
objects(>1G), I get error of "503 service unavailable"(on swift cli) after
a long wait.

I dig into logs and found that none of the replicator services are running.
All of them give the following error(array index out of range)

Mar  8 12:42:03 yahoo-128 account-replicator Beginning replication run
Mar  8 12:42:03 yahoo-128 account-replicator STDOUT: Traceback (most recent
call last):
Mar  8 12:42:03 yahoo-128 account-replicator STDOUT: File
"/usr/lib/python2.7/dist-packages/eventlet/greenpool.py", line 80, in
_spawn_n_impl
Mar  8 12:42:03 yahoo-128 account-replicator STDOUT: func(*args, **kwargs)
Mar  8 12:42:03 yahoo-128 account-replicator STDOUT: File
"/usr/lib/python2.7/dist-packages/swift/common/db_replicator.py", line 398,
in _replicate_object
Mar  8 12:42:03 yahoo-128 account-replicator STDOUT: nodes =
self.ring.get_part_nodes(int(partition))
Mar  8 12:42:03 yahoo-128 account-replicator STDOUT: File
"/usr/lib/python2.7/dist-packages/swift/common/ring/ring.py", line 201, in
get_part_nodes
Mar  8 12:42:03 yahoo-128 account-replicator STDOUT: if not (r[part] in
seen_ids or seen_ids.add(r[part]))]
Mar  8 12:42:03 yahoo-128 account-replicator STDOUT: IndexError: array
index out of range
Mar  8 12:42:03 yahoo-128 account-replicator STDOUT: Traceback (most recent
call last):
Mar  8 12:42:03 yahoo-128 account-replicator STDOUT: File
"/usr/lib/python2.7/dist-packages/eventlet/greenpool.py", line 80, in
_spawn_n_impl
Mar  8 12:42:03 yahoo-128 account-replicator STDOUT: func(*args, **kwargs)
Mar  8 12:42:03 yahoo-128 account-replicator STDOUT: File
"/usr/lib/python2.7/dist-packages/swift/common/db_replicator.py", line 398,
in _replicate_object
Mar  8 12:42:03 yahoo-128 account-replicator STDOUT: nodes =
self.ring.get_part_nodes(int(partition))
Mar  8 12:42:03 yahoo-128 account-replicator STDOUT: File
"/usr/lib/python2.7/dist-packages/swift/common/ring/ring.py", line 201, in
get_part_nodes
Mar  8 12:42:03 yahoo-128 account-replicator STDOUT: if not (r[part] in
seen_ids or seen_ids.add(r[part]))]
Mar  8 12:42:03 yahoo-128 account-replicator STDOUT: IndexError: array
index out of range
Mar  8 12:42:03 yahoo-128 account-replicator Replication run OVER
Mar  8 12:42:03 yahoo-128 account-replicator Attempted to replicate 3 dbs
in 0.02536 seconds (118.26332/s)
Mar  8 12:42:03 yahoo-128 account-replicator Removed 0 dbs
Mar  8 12:42:03 yahoo-128 account-replicator 1 successes, 0 failures
Mar  8 12:42:03 yahoo-128 account-replicator no_change:1 ts_repl:0 diff:0
rsync:0 diff_capped:0 hashmatch:0 empty:0

Additionally, while the transfer of a big file was going on(4GB), I got the
following error in logs.

proxy-server ERROR with Object server 10.0.1.1:6000/sdb1 re: Trying to
write to
/v1/AUTH_cccc28c949f34bd3b5fb295cffb69ed4/data_dir/data_dir/Feb27/PRIVATE/AVCHD/BDMV/STREAM/00002.MTS:
ChunkWriteTimeout (10s)

Any suggestions?

--
Shashank Sahni