maria-discuss team mailing list archive

[Walter Weinmann <walter.at.konnexions@xxxxxxxxx>]

Hi Daniel

The original goal of this project was the generation of large amounts of
anonymous and random data for relational database systems useful e.g. for
stress testing.

I am really pleased with your reaction to my announcement of DBSeeder
version 2.9.0. I am now working on version 2.9.1 (see here:
https://github.com/KonnexionsGmbH/db_seeder/tree/wwe_2.9.1) and will try to
incorporate as much of your suggestions and issue reports as possible.

*1. Compilation with OpenJDK 16*

Unfortunately, I can't reproduce the problem. I am also of the opinion that
I run OpenJDK 16 both under Windows 10:

*openjdk version "16.0.1" 2021-04-20*
*OpenJDK Runtime Environment (build 16.0.1+9-24)*
*OpenJDK 64-Bit Server VM (build 16.0.1+9-24, mixed mode, sharing)*


as well as under Ubuntu 20.04:

*openjdk version "16.0.1" 2021-04-20*
*OpenJDK Runtime Environment (build 16.0.1+9-24)*
*OpenJDK 64-Bit Server VM (build 16.0.1+9-24, mixed mode, sharing)*


The compilation should be done in the root directory of the db_seeder
repository with the following command:

*gradle copyJarToLib*



*2. Docker Image Size*

I created a custom image for the development and operation of DBSeeder. The
size of the image is now only 1.73 GB. Details can be found here:
https://hub.docker.com/repository/docker/konnexionsgmbh/db_seeder.


*3. README.md*

I have restructured this document on your suggestion with the intention to
present DBSeeder more clearly.


*4. Docker Compose*

I removed Docker Compose, which doesn't make much sense in combination with
DBSeeder.


*5. line 13 in script run_db_seeder.sh*

I have corrected this issue.


*6. Login Docker Hub*

I removed the unnecessary login to Docker Hub. It seems that at some point
I misinterpreted the problems described here:
https://github.com/docker/hub-feedback/issues/1103.


*7. Defaults in script run_db_seeder_single.sh*

*run_db_seeder_single* is just an auxiliary script used by the script
*run_db_seeder* and is not called directly (see: README.md diagram in
section 4.1.2). Therefore the script variables in *run_db_seeder_single*
have no default values unlike in *run_db_seeder*. The easiest way to
customize these environment variables is directly in the *run_db_seeder*
script.


*8. Feature request*

I added the following two environment variables in the *run_db_seeder*
script specifically for MariaDB:

    DB_SEEDER_CHARACTER_SET_SERVER=utf8mb4
    DB_SEEDER_COLLATION_SERVER=utf8mb4_unicode_ci


*9. MDEV-25434*

When this extension is available I will gladly switch to health check on
MariaDB (similar to Apache Derby for example).


*10. Podman*

I think there is no need for action on the part of DBSeeder?


*11. JSON output*

Here the effort seems to me too large, since the realization would have to
take place DBMS specifically and this would go beyond the bounds with the
currently 24 supported systems. In this context DBSeeder offers two
functionalities:

1. the database schema is generated from a JSON file - see in README.md
section 2 and the files in the *resources/json* directory.
2. runtime statistics are produced in tsv format - see README.md section
4.4 and files in *resources/statistics* directory

---

I hope that I have understood your suggestions and objections correctly and
have offered satisfactory solutions. If you notice anything else then just
let me know.

I plan to release version 2.9.1 by the end of June at the latest. It will
also include all updates of the used DBMS Docker images and JDBC drivers.

Thanks again and best regards

Walter







On Mon, 7 Jun 2021 at 05:50, Daniel Black <daniel@xxxxxxxxxxx> wrote:

>
> Thanks Walter for letting everyone know.
>
> I did play around with it as it might be useful in attempting to replicate
> sample data for users experiencing problems.
>
> What was your intended purpose when creating this?
>
>
> Suggestions/minor problems that I came up with testing it:
>
> It doesn't compile with openjdk and I needed to install oraclejdk to test
> this. This wasn't documented (except .travis.yml that tipped me off).
>
> docker.io/konnexionsgmbh/kxn_dev is a very big image - 7.56G
>
> README.md
> - the database list in section 1  - was moved into the 1.1 table this
> would be easier to read.
> - Chines typo
> - in reference to docker-compose.yml needs to document creating Transfer
> directory
>
> run_db_seeder.sh:
>
> line 13 - is this meant to be "rm -rf $PWD/tmp/* || sudo rm -rf
> $PWD/tmp/*"?
>
> the docker username/password seem to be optional
>
> It's unclear what username/password this is referring to in the output:
> Username:
> Password:
> Login Succeeded!
>
> ================================================================================
> Start ./scripts/run_db_seeder_single.sh
>
> suggest using shell default values allowing user to override:
> export DB_SEEDER_CONNECTION_PORT=${DB_SEEDER_CONNECTION_PORT:-3306}
>
> Feature request, can there be an environment way to specify
> arguments/settings to the container (defaulting to utf8?)?
> --character-set-server=utf8mb4 --collation-server=utf8mb4_unicode_ci (is
> this required?)
>
> src/main/java/ch/konnexions/db_seeder/jdbc/mariadb/MariadbSeeder.java the
> initialization could be replaced with:
> diff --git a/scripts/run_db_seeder_setup_dbms.sh
> b/scripts/run_db_seeder_setup_dbms.sh
> index 096bc61..c47cbab 100755
> --- a/scripts/run_db_seeder_setup_dbms.sh
> +++ b/scripts/run_db_seeder_setup_dbms.sh
> @@ -387,18 +387,21 @@ if [ "${DB_SEEDER_DBMS_DB}" = "mariadb" ]; then
>      echo "Docker create db_seeder_db (MariaDB Server
> ${DB_SEEDER_VERSION})"
>
>      docker network create db_seeder_net  2>/dev/null || true
> -    docker create -e        MYSQL_ROOT_PASSWORD=mariadb \
> +    docker create -e
>  MARIADB_ROOT_PASSWORD="${DB_SEEDER_PASSWORD_SYS}" \
> +                 -e        MARIADB_DATABASE="${DB_SEEDER_DATABASE}" \
> +                 -e        MARIADB_USER="${DB_SEEDER_USER}" \
> +                 -e        MARIADB_PASSWORD="${DB_SEEDER_PASSWORD}" \
>                    --name    db_seeder_db \
>                    --network db_seeder_net \
>                    -p
>  "${DB_SEEDER_CONNECTION_PORT}":"${DB_SEEDER_CONTAINER_PORT}"/tcp \
> -                  mariadb:"${DB_SEEDER_VERSION}"
> --character-set-server=utf8mb4 --collation-server=utf8mb4_unicode_ci
> +                  mariadb:"${DB_SEEDER_VERSION}"
> --character-set-server=utf8mb4 --collation-server=utf8mb4_unicode_ci
> --port="${DB_SEEDER_CONNECTION_PORT}"
>
>      echo "Docker start db_seeder_db (MariaDB Server ${DB_SEEDER_VERSION})
> ..."
>      if ! docker start db_seeder_db; then
>          exit 255
>      fi
>
>     sleep 30
>
> Once MDEV-25434 is done I hope the sleep 30 can be replaced. If there's
> any other MariaDB container features needed to work better please let me
> know (https://github.com/MariaDB/mariadb-docker/issues or
> https://jira.mariadb.org component Docker).
>
> Testing with podman instead of docker with a alias docker=podman worked
> quite well too.
>
> A useful feature would be a SQL statement, probably over
> information_schema and maybe table statistics columns that generates the
> JSON output from an existing schema.
>
> Thanks for sharing
>
>
> On Sun, Jun 6, 2021 at 1:58 AM Walter Weinmann <
> walter.at.konnexions@xxxxxxxxx> wrote:
>
>> We are happy to announce that version 2.9.0 of DBSeeder (
>> https://github.com/KonnexionsGmbH/db_seeder) is now available for
>> general use. DBSeeder also supports the latest version of MariaDB Server
>> available on DockerHub (10.6.1).
>> _______________________________________________
>> Mailing list: https://launchpad.net/~maria-discuss
>> Post to     : maria-discuss@xxxxxxxxxxxxxxxxxxx
>> Unsubscribe : https://launchpad.net/~maria-discuss
>> More help   : https://help.launchpad.net/ListHelp
>>
>