dhis2-users team mailing list archive
-
dhis2-users team
-
Mailing list archive
-
Message #13753
Re: Major speed issue with pg_restore
-
To:
Bob Jolliffe <bobjolliffe@xxxxxxxxx>
-
From:
Edward Robinson <erobinson@xxxxxxxxxxxxxxxxxx>
-
Date:
Wed, 19 Jul 2017 14:16:41 +0000
-
Accept-language:
en-US
-
Authentication-results:
gmail.com; dkim=none (message not signed) header.d=none;gmail.com; dmarc=none action=none header.from=projectbalance.com;
-
Cc:
dhis2-users <dhis2-users@xxxxxxxxxxxxxxxxxxx>
-
In-reply-to:
<CACd=f9dg8OOvXPb1G-oBCk5KR9tOYvHJimzs1j49M=S45BG30g@mail.gmail.com>
-
Spamdiagnosticmetadata:
NSPM
-
Spamdiagnosticoutput:
1:99
-
Thread-index:
AdL8Qv0hVdlHNLGQToSpRSLZ4Y1GzQAVhWAAAAArzwAAAAUWoAA9USlQAANvnoAANaxaoAAhmuYAAAd0eCAABNLIAAAzFeIgAAdhSoAAILFucA==
-
Thread-topic:
[Dhis2-users] Major speed issue with pg_restore
Agree, the system has two independent disks, I hadn’t set up any mirroring or striping – I had basically just setup the OS in both cases and performed OS updates. Pretty much a vanilla config on both. My experience is that if the SATA channel drivers aren’t properly loaded, the disks will work in legacy mode which slows things down considerably since no interrupts are used. In the old days with the IDE channel, it was interrupt 14 and 15 for channel 1 and 2 but that’s no longer the case with SATA. To be honest I haven’t come across this before in Linux since most good distros have all the software required to run all but the newest hardware and given that these machines are a few years old, I would assume they are supported (but I imagine this may be something worth troubleshooting). Ultimately I do think online hosting for this project will probably be the best given the difficulty supporting any hardware in this location so we are exploring those options.
Some things I hadn’t tried yet:
https://serverfault.com/questions/790723/postgres-9-5-restoring-dumps-is-very-very-slow
“
It does not seem to be a PG problem. More likely it's a language or locale problem with PG on Ubuntu 16.04, because installing huge indexed tables with foreign languages and therefore with many utf-8 chars cause the problem. Do we miss a language package? On another server with Ubuntu 14.04 everything works fine.
“
It’s not this, since I had dumped using the default copy method:
http://manpages.ubuntu.com/manpages/xenial/man1/pg_dump.1.html
“--column-inserts
--attribute-inserts
Dump data as INSERT commands with explicit column names (INSERT
INTO table (column, ...) VALUES ...). This will make restoration
very slow; it is mainly useful for making dumps that can be loaded
into non-PostgreSQL databases. However, since this option generates
a separate command for each row, an error in reloading a row causes
only that row to be lost rather than the entire table contents.
--inserts
Dump data as INSERT commands (rather than COPY). This will make
restoration very slow; it is mainly useful for making dumps that
can be loaded into non-PostgreSQL databases. However, since this
option generates a separate command for each row, an error in
reloading a row causes only that row to be lost rather than the
entire table contents. Note that the restore might fail altogether
if you have rearranged column order. The --column-inserts option is
safe against column order changes, though even slower.
“
References
-
Major speed issue with pg_restore
From: Edward Robinson, 2017-07-14
-
Re: Major speed issue with pg_restore
From: Bob Jolliffe, 2017-07-14
-
Re: Major speed issue with pg_restore
From: Bob Jolliffe, 2017-07-14
-
Re: Major speed issue with pg_restore
From: Edward Robinson, 2017-07-14
-
Re: Major speed issue with pg_restore
From: Edward Robinson, 2017-07-15
-
Re: Major speed issue with pg_restore
From: Bob Jolliffe, 2017-07-15
-
Re: Major speed issue with pg_restore
From: Edward Robinson, 2017-07-16
-
Re: Major speed issue with pg_restore
From: Lars Helge Øverland, 2017-07-17
-
Re: Major speed issue with pg_restore
From: Edward Robinson, 2017-07-17
-
Re: Major speed issue with pg_restore
From: Bob Jolliffe, 2017-07-17
-
Re: Major speed issue with pg_restore
From: Edward Robinson, 2017-07-18
-
Re: Major speed issue with pg_restore
From: Bob Jolliffe, 2017-07-18