mirror of
https://github.com/iiab/iiab.git
synced 2025-02-15 04:32:11 +00:00
Delete internetarchive/README.rst as README.md is sufficient
This commit is contained in:
A Holt
GitHub
parent
3164a55fae
commit
49c606e638
1 changed files with 0 additions and 231 deletions
|
|
@ -1,231 +0,0 @@
|
|||
Internet Archive - Universal Library project README
|
||||
===================================================
|
||||
|
||||
This Ansible role installs the Internet Archive’s dweb-mirror project on
|
||||
Internet-in-a-Box (IIAB).
|
||||
|
||||
The project is a local server that allows users to browse resources from
|
||||
the Internet Archive stored on local drives - including USB drives.
|
||||
|
||||
It includes a crawler that can regularly synchronize local collections,
|
||||
against a list of Internet Archive items and collections, and those
|
||||
collections can be moved between installations.
|
||||
|
||||
When connected to the internet, the server works as a Proxy, i.e. it
|
||||
will store Internet Archive content the user views for later off-line
|
||||
viewing.
|
||||
|
||||
There are components to integrate the IA server with decentralized tools
|
||||
including IPFS, WebTorrent, GUN, WOLK, both for fetching content and for
|
||||
serving it back to the net or locally.
|
||||
|
||||
This is an ongoing project, continually adding support for new Internet
|
||||
Archive content types; new platforms; and new decentralized transports.
|
||||
|
||||
Using it
|
||||
--------
|
||||
|
||||
Starting server
|
||||
~~~~~~~~~~~~~~~
|
||||
|
||||
| The server is started and restarted automatically.
|
||||
| It can be turned on or off at a terminal window with
|
||||
``service internetarchive start`` or ``service internetarchive stop``
|
||||
|
||||
Browsing
|
||||
~~~~~~~~
|
||||
|
||||
The server can be accessed at [http://box.lan:4244] (try
|
||||
[http://box.local:4244] if that doesn’t work)
|
||||
|
||||
If you don’t get a Archive UI then look at the server log (in browser
|
||||
console) to see for any “FAILING” log lines which indicate a problem.
|
||||
|
||||
Expect to see errors in the Browser log for
|
||||
``http://localhost:5001/api/v0/version?stream-channels=true`` which is
|
||||
checking for a local IPFS server which is not started here.
|
||||
|
||||
Expect, on slower machines or slower network connections, to see no
|
||||
images the first time, refresh after a little while and most should
|
||||
appear.
|
||||
|
||||
Administration
|
||||
--------------
|
||||
|
||||
Administration is carried out through the same User Interface as
|
||||
browsing.
|
||||
|
||||
Access [http://localhost:4244/local] to see a display of local content,
|
||||
this interface is under development and various admin tools will be
|
||||
added here. *at some point this will become the default page*.
|
||||
|
||||
Access [http://localhost:4244] to get the Internet Archive main
|
||||
interface if connected to the net.
|
||||
|
||||
While viewing an item or collection, the “Crawl” button in the top bar
|
||||
indicates whether the item is being crawled or not. Clicking it will
|
||||
cycle through three levels: \* No crawling \* Details - sufficient
|
||||
information will be crawled to display the page, for a collection this
|
||||
also means getting the thumbnails and metadata for the top items. \*
|
||||
Full - crawls everything on the item, this can be a LOT of data,
|
||||
including full size videos etc, so use with care if bandwidth/disk is
|
||||
limited.
|
||||
|
||||
Disks
|
||||
~~~~~
|
||||
|
||||
The server checks for disks in all the likely places, the list of places
|
||||
it checks, in an unmodified installation can be seen at
|
||||
``https://github.com/internetarchive/dweb-mirror/blob/master/configDefaults.yaml#L7``
|
||||
|
||||
This includes top level directories on usb drives at
|
||||
``/media/pi/*/archiveorg``
|
||||
|
||||
You can override this in ``dweb-mirror.config.yaml`` in the home
|
||||
directory of the user that runs the server, this is currently
|
||||
``/root/dweb-mirror.config.yaml`` (see ‘Advanced’ below )
|
||||
|
||||
Items are stored in subdirectories of the first of these directories
|
||||
found, but read from any of the locations.
|
||||
|
||||
If you disk space is getting full, its perfectly safe to delete any
|
||||
subdirectories, except the ``.hashstore`` at the top level of each, the
|
||||
server will refetch what it needs if you browse to the item again when
|
||||
connected to the internet.
|
||||
|
||||
Maintenance
|
||||
~~~~~~~~~~~
|
||||
|
||||
If you are worried about corruption, or after for example hand-editing
|
||||
or moving cached items around.
|
||||
|
||||
::
|
||||
|
||||
# Run everything as root
|
||||
sudo sh
|
||||
# cd into location for your installation
|
||||
cd /opt/iiab/internetarchive/node_modules/@internetarchive/dweb-mirror
|
||||
./internetarchive -m
|
||||
|
||||
This will usually take about 5-10 minutes depending on the amount of
|
||||
material cached, just to rebuild a table of checksums.
|
||||
|
||||
Advanced
|
||||
~~~~~~~~
|
||||
|
||||
Most functionality of the tool is controlled by two YAML files, the
|
||||
second of which you can edit if you have access to the shell.
|
||||
|
||||
You can view the current configuration by going to
|
||||
[http://box.lan:4244/info] or [http://localhost:4244/info] depending on
|
||||
how you are connected.
|
||||
|
||||
The default, and user configurations are displayed as the ``0`` and
|
||||
``1`` item in the ``/info`` call.
|
||||
|
||||
In the Repo is a `default YAML
|
||||
file <https://github.com/internetarchive/dweb-mirror/blob/master/configDefaults.yaml>`__
|
||||
which is commented. It would be a bad idea to edit this, so I’m not
|
||||
going to tell you where it is on your installation! But anything from
|
||||
this file can be overridden by lines in
|
||||
``/root/dweb-mirror.config.yaml``. Make sure you understand how yaml
|
||||
works before editing this file, if you break it, you can copy a new
|
||||
default from `dweb-mirror.config.yaml on the
|
||||
repo <https://github.com/internetarchive/dweb-mirror/blob/master/configDefaults.yaml#L7>`__
|
||||
|
||||
TODO Note this file will probably move location.
|
||||
|
||||
Note that this file is also edited automatically when the Crawl button
|
||||
described above is clicked.
|
||||
|
||||
As the project develops, this file will be editable via a UI.
|
||||
|
||||
Update
|
||||
------
|
||||
|
||||
Dweb-mirror is under rapid development, as is the Javascript UI. Its
|
||||
recommended to update frequently.
|
||||
|
||||
From a Terminal window
|
||||
|
||||
::
|
||||
|
||||
sudo sh # Run all commands as root
|
||||
cd /opt/iiab/internetarchive
|
||||
yarn upgrade # Currently this can take up to about 20 minutes to run, we hope to reduce that time
|
||||
|
||||
Crawling
|
||||
--------
|
||||
|
||||
The Crawler will be built into the UI fairly soon, for now it has to be
|
||||
run in a terminal window.
|
||||
|
||||
Its highly configurable either through the YAML file described above, or
|
||||
from the command line.
|
||||
|
||||
In a shell
|
||||
|
||||
::
|
||||
|
||||
# Run all commands as root from dweb-mirror's directory
|
||||
sudo sh
|
||||
|
||||
# cd into location for your installation
|
||||
cd /opt/iiab/internetarchive/node_modules/@internetarchive/dweb-mirror
|
||||
|
||||
# To get a full list of possible arguments
|
||||
./internetarchive --help
|
||||
|
||||
# Perform a standard crawl
|
||||
./internetarchive --crawl
|
||||
|
||||
# To fetch the "foobar" item from IA.
|
||||
./internetarchive --crawl foobar
|
||||
|
||||
# To crawl top 10 items in the prelinger collection sufficiently to display and put
|
||||
# them on a disk plugged into the /media/pi/xyz
|
||||
# TODO check where pi actually put them.
|
||||
./internetarchive --copydirectory /media/pi/xyz/archiveorg --crawl --rows 10 --level details prelinger
|
||||
|
||||
Troubleshooting
|
||||
---------------
|
||||
|
||||
There are two logs of relevance, the browser and the server.
|
||||
|
||||
**Browser**: If using Chrome then this is at View / Developer Tools /
|
||||
Javascript Console or something similar.
|
||||
|
||||
**Server**: From a Terminal window.
|
||||
|
||||
::
|
||||
|
||||
journalctl -u internetarchive
|
||||
|
||||
Known Issues
|
||||
------------
|
||||
|
||||
See `github dweb-mirror
|
||||
issues <https://github.com/internetarchive/dweb-mirror/issues>`__; and
|
||||
`github dweb-archive
|
||||
issues <https://github.com/internetarchive/dweb-archive/issues>`__;
|
||||
|
||||
More info
|
||||
---------
|
||||
|
||||
Dweb-Mirror Lives on github at:
|
||||
`dweb-mirror <https://github.com/internetarchive/dweb-mirror>`__;
|
||||
`source <https://github.com/internetarchive/dweb-mirror>`__;
|
||||
`issues <https://github.com/internetarchive/dweb-mirror/issues>`__;
|
||||
`API.md <./API.md>`__ API documentation for dweb-mirror
|
||||
|
||||
This project is part of our larger Dweb project, see also:
|
||||
`dweb-universal <https://github.com/internetarchive/dweb-universal>`__
|
||||
info about others distributing the web;
|
||||
`dweb-transport <https://github.com/internetarchive/dweb-transport>`__
|
||||
miscellaneous inc GUN gateway and webtorrent;
|
||||
`dweb-objects <https://github.com/internetarchive/dweb-objects>`__
|
||||
library of dweb objects;
|
||||
`dweb-archive <https://github.com/internetarchive/dweb-archive>`__
|
||||
archive UI in Javascript;
|
||||
`dweb-archivecontroller <https://github.com/internetarchive/dweb-archive>`__
|
||||
Knows about the structure of archive objects;
|
||||
Loading…
Reference in a new issue