← Back to team overview

zim-wiki team mailing list archive

Directly adding files and sub-directories and re-indexing

 

Hi Jaap,


A personal thanks for Zim - an excellent tool.

Also a congratulation for the quality of the code and sapience shown.

This message I'm sending is very long and may be not that much important to
the community.

So, if for any reason you aren't able to read it, please rest assured that
it is perfectly OK and that I am and will be grateful in any circumstance.

But, if you manage to read it, I would very much appreciate any comment or
suggestion you may find pertinent.

Also, please accept apologizes for my "english" which is not my
mother-language.



I do a lot of researching on Internet and I am used to save almost
everything that found useful in the process; not only links but also .mht
files and PDF prints of pages consulted and others related files.

Saving is needed  not only because of offline reading but, most
importantly, to rest assured that consulted information will stay available
for later usage and reference.

So, I am using Zim almost like Zotero (http://www.zotero.org/).

The process I am using on Zim is rather time consuming done manually - due
to quantity - and is error prone.

If filenames are not "normalized" as Zim expects, they don't get indexed.
Additionally, if directories are "created manually" that information is not
seen.
Also, this can create general problems with indexing, making Zim inoperable
because it detects these index-related problems. When this happens a
"normalization" of all affecting paths/filenames is required to be done
manually so that Zim will work again.

I would like to automate this process as much as possible.

I am doing and considering the following:

1) Put downloaded pages grouped by matters searched on related directories
inside a general base drectory.
    E.g.: /data/z3l/zim-wiki/Downloads-to-Zim

2) by making a simple change to the indexed functionality of Zim it outputs
the fullpath of invalid files (and paths) when performing a full index.

    ./zim.py --index  /data/z3l/zim-wiki/ &> files_list_invalid.txt
(see Listing 1, below)

2.1) I tried to add an external option, such as --indexpaths but have not
been successful (without considering a global variable) as I couldn't get
this option information way to the function zim/stores/
files.py/get_pagelist(self, path) where warnings are issued by writing to
logger.warn() when appropriate.
2.1.1) I will try to use a different approach later.

3) I will read the created file (files_list_invalid.txt) and:
3.1) "normalize" the collected invalid paths as per Zim requirements:
    * change '  ' to '_'
    * skip "*.txt" files (only to be done if there isn't a corresponding
directory, as .txt may need to be attached)
    * skip ".*" and "_*" directories
    * others that may be needed but I haven't found out yet

    Note: this is now done "externally", that is, a python script is run
and changes are made on filesystem.

4) the idea is then to run a script that:
4.1) moves the files to Notebooks' filesystem tree
4.2) moves directories to Notebooks' filesystem tree
4.2.1) creates a Zim text document ".txt" with the name of the directory,
so that the directory may be indexed.
4.2.2) on the Zim page, create links to existing files inside the directory
4.2.3) repeat for each directory and recurse

4.3) pages will have a location derived from the directory name which
contains tag information on it.
Eg.:
    * the content of directory named :
      @Python,@Papers,@Storage;@Programming,@Dedupliction;esFS - Storage
Efficient Filesystem in Python

      will be saved on the following partially-existent (or to be created)
path:

..../Python/Papers/Storage/FS_-_Storage_Efficient_Filesystem_in_Python/

      The tags following the first ';' are to be added to the
corresponding  .txt file to be created.
     * the file FS_-_Storage_Efficient_Filesystem _in_Python.txt will be
created on the parent dir
     * links to files existing on directory
FS_-_Storage_Efficient_Filesystem _in_Python will be inserted on the .txt
file.
Eg.:
     FS_-_Storage_Efficient_Filesystem_in_Python.txt:
     -------Begin-------
            Wiki-Format: zim 0.4
            Creation-Date: 2012-10-18T17:41:13+01:00

            ====== FS - Storage Efficient Filesystem in Python ======
            Created Quinta 18 Outubro 2012

            [[./244-941-1-PB.pdf]]

            [[./FUSE’ing_Python_for_Development_244.mht]]

            @Python
            @Papers
            @Storage
            @Programming
            @Deduplication
     -------End-------
     * parent '.txt' files  may need to be actualized.

4.4) perform a Zim --index

5) Later, when more familiar with Zim innards, I would like to:
5.1) perform part of the above operations with the own Zim existing
functionalities (fs.py, ...)
5.2) arrange a way to perform the actions wanted with a greater integration
with Zim. For example:
      * Zim plug-in for firefox would be nice, if some features were added
      * make Zim (optionally) accept directly saved files and create
directories, making him "normalize" paths/filenames and create
corresponding .txt page files for directories.

Listings:
(1)
z2@pclevo:~/Development/python/zim-0.57$ ./zim.py --index
/data/z3l/zim-wiki
WARNING: "/data/z3l/zim-wiki/Downloads-to-Zim/@PDF,@PDF_Tools,@Okular;PDF
editor annotations_Okular"
WARNING:
"/data/z3l/zim-wiki/Downloads-to-Zim/@Python,@Papers,@Storage;@Programming,@Dedupliction;esFS
- Storage Efficient Filesystem in Python"
WARNING: "/data/z3l/zim-wiki/Downloads-to-Zim/@Python,@Papers;Python Papers"
WARNING:
"/data/z3l/zim-wiki/Downloads-to-Zim/@Python,@Programming,@Testing;@Scripts,@Testing,@Examples,Tests
in Python"
WARNING:
"/data/z3l/zim-wiki/Downloads-to-Zim/@Python,@Scripts,@Shell_tools,@Programming,@Examples,Python
Shell Utilities"
WARNING: "/data/z3l/zim-wiki/Downloads-to-Zim/@python,@lambda Python Lamba
Funcrions
...
(these were saved directly inside Zim's tree filesystem)
WARNING:
"/data/z3l/zim-wiki/Notebooks/Notes/Home/Development/Python/TUI/Python
EasyGUI"
WARNING:
"/data/z3l/zim-wiki/Notebooks/Notes/Home/Development/Python/TUI/UniCurses
for Python"
WARNING:
"/data/z3l/zim-wiki/Notebooks/Notes/Home/Development/Python/TUI/Urwid -
Console User Interface Library"
WARNING:
"/data/z3l/zim-wiki/Notebooks/Notes/Home/Development/Python/TUI/ranger -
file manager"
WARNIN ...


If you have any considerations, advises, or just your raw opinion about the
usefulness and feasibility, they will be much appreciated.

Meanwhile, please accept my

Very best regards,

Jose Lourenco
_______________________________________________________________
Para manter a Wikipedia a funcionar por favor considere fazer a sua doação.
Please consider donating to maintain Wikipedia alive.
 Support Wikipedia<http://wikimediafoundation.org/wiki/Support_Wikipedia/en>
 _______________________________________________________________
http://jal.stumbleupon.com
http://www.lourenco.ws/about-me

Follow ups