InvenioRDM Alpha 1 (October release)

Projects Invenio RDM
1

We’re happy to announce the very first alpha release of InvenioRDM.

What should you expect?

Be aware that you will not see anything that remotely resembles Zenodo! There has been no work on the User Interface. Everything is very barebones right now.

The primary focus has been on getting the basic modules bootstrapped and a first version of the installation experience.

You will do the following:

  1. Create, install and setup your first development InvenioRDM instance with a new Invenio CLI tool.
  2. Customize a logo

What do you need to do?

  1. Leave all feedback as comments to this thread using the template below.
  2. Report any bugs you experience (simply reply with comments to this thread).
  3. Report a) what worked well and b) what did not work well (comments can be related to anything not only the software).
  4. Report what you would expect to see documented in the documentation and what you would like to know more about.

You can use the following template:

## Bugs

## What worked well

## What didn't work well

## Wishes for documentation

Install (TL;DR)

For the impatient :slight_smile: Execute the following commands:

pip install invenio-cli
invenio-cli --flavour=RDM init
invenio-cli build --dev --pre --lock
invenio-cli run --dev --bg
invenio-cli setup --dev
pipenv run invenio rdm-records demo

Afterwards, you should be able to see something like this on https://127.0.0.1:5000/

31
Screenshot 2019-11-08 at 17.16.31.png1209×540 22.8 KB

59
Screenshot 2019-11-08 at 17.21.59.png1171×926 56.2 KB

01
Screenshot 2019-11-08 at 18.33.01.png1209×945 50 KB

Install (the long version)

See https://github.com/inveniosoftware/invenio-app-rdm/wiki

To see the logo customization, please view this long version.

Known issues

See https://github.com/inveniosoftware/invenio-app-rdm/wiki

Troubleshooting

We will update this as we get information during next week.

  1. Stop all existing running docker containers (otherwise, you may have port conflicts): docker stop \$(docker ps -a -q)
2

Bugs

  • No python 3.7 support. Using an unsupported python version flags a warning about python not found but it’s not the most descriptive.
  • dependency cchardet on a Mac using conda has compilation issues. Workaround is setting the environment variable MACOSX_DEPLOYMENT_TARGET=10.9, and could be solved by sending a PR like https://github.com/pandas-dev/pandas/pull/24274/files
  • Use of realpath in scripts/bootstrap means that coreutils needs to be installed (available from brew for mac)

And one I haven’t figured out:

  • invenio.cfg in the virtualenv seems to be a symbolic link to itself (which then breaks things). It’s probably a Mac issue.

    /Users/tmorrell/.local/share/virtualenvs/caltech-test-JKcZowdS/var/instance
    (base) notebook-tm:instance tmorrell$ ls -l
    total 0
    drwxr-xr-x 12 tmorrell staff 384 Nov 8 15:46 assets
    lrwxr-xr-x 1 tmorrell staff 87 Nov 8 15:46 invenio.cfg -> /Users/tmorrell/.local/share/virtualenvs/caltech-test-JKcZowdS/var/instance/invenio.cfg
    drwxr-xr-x 12 tmorrell staff 384 Nov 8 15:46 static

1 Like
3

Bugs

I had docker and ptyhon 3.7 on a fairly plain install of OS X Catalina. Most of the setup went well. But running the four steps above, without changing anything, required me to name my site Invenio App RDM so that the db connection worked. Otherwise I would get

sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) FATAL:  password authentication failed for user "invenio-app-rdm"

Also, selecting sqlite or mysql does not change the config either.

I guess this is much the same as Tom experienced.

After setting the correct demo site name, I was also able to load demo records.

At the end, the server seemed to run fine, but I was not able to connect to port 5000:

$ curl localhost:5000
curl: (56) Recv failure: Connection reset by peer

What worked well

The way it handles docker is pretty neat for quick testing/setup.

What didn’t work well

When stating that 4 commands give you a working site, it just has to work. :wink:

Wishes for documentation

I installled invenio-cli in an virtual environment. Without browsing code, it is a bit hard to know what is going on. It looks like the invenio-cli to some degree is using pipenv automatically? I got some Courtesy notices about it. Also the way it manages docker containers would make sense to briefly comment on/document.

4

Hi Audun,

Thanks a lot for the feedback. I have one quick question: The curl, could you try over HTTPS. I guess we should have specified/give the option to run both.

If still not working, is there any stack trace that could be provided?

Thanks once again!
Pablo

5

Bugs

Seems to me, that sometimes the flavor is obligatory:

invenio-cli build --dev --pre --lock
ERROR:root:No flavour specified.
invenio-cli build --flavour=RDM --dev --pre --lock # this worked

Didn’t notice, that “run” was actually running the server and blocking the terminal, perhaps as suggestion:

invenio-cli run --dev --bg &

As stated above, I also got the following notice:

Courtesy Notice: Pipenv found itself running within a virtual environment, so it will automatically use that environment, instead of creating its own for any project. You can set PIPENV_IGNORE_VIRTUALENVS=1 to force pipenv to ignore that environment and create its own instead. You can set PIPENV_VERBOSITY=-1 to suppress this warning.

What worked well

Quick installation. Maybe you should consider some special setups like a venv for the QuickStartGuide.

What didn’t work well

  1. My es Container (docker.elastic.co/elasticsearch/elasticsearch-oss:6.8.2) is marked as “unhealty”. Don’t know if this log entries have something to do with it:

[2019-11-13T15:55:39,741][WARN ][o.e.d.r.a.a.i.RestPutMappingAction] [jCZKmDc] [types removal] The parameter include_type_name should be explicitly specified in create index requests to prepare for 7.0. In 7.0 include_type_name will default to ‘false’, and requests are expected to omit the type name in mapping definitions.>

[2019-11-13T15:55:39,763][WARN ][o.e.d.c.m.MetaDataCreateIndexService] [jCZKmDc] the default number of shards will change from [5] to [1] in 7.0.0; if you wish to continue using the default of [5] shards, you must manage this on the create index request or with an index template

[2019-11-13T15:55:39,848][INFO ][o.e.c.m.MetaDataCreateIndexService] [jCZKmDc] [records-record-v1.0.0-1573660539] creating index, cause [api], templates , shards [5]/[1], mappings [record-v1.0.0]

  1. Stop your local postgresql, if running on 5432. Bear that in mind :slight_smile:

Wishes for documentation

In the end, I’d really like detailed information about production state. What do I need to configure in order to use SSL/certificates? Which envvars have to be set? etc.

6 
**Bugs**
Not really a bug: I had to call sudo pip instead of pip.

**What worked well**
After I figured out some misconfigurations of my servers all went very well, really. I was working with CentOS 7 and CentOS8.

## What didn't work well
It just worked as expected.

## Wishes for documentation
Setting up docker in CentOS 8 is a mess. I'll provide some documentation for it.
7

Bugs

After installation, doing pipenv run invenio rdm-records demo fails due to a elasticsearch.exceptions.AuthorizationException: AuthorizationException(403, 'cluster_block_exception', 'index [records-record-v1.0.0-1573830067] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];')

What worked well

Once I got my Python environment in order (thanks, @egabancho), the install worked as expected.

What didn’t work well

I had to get my Python environment in order (…), see Bugs

Wishes for documentation

A few words on what is needed python-wise (like python 3.x.x, dev etc.)

8

Hello all,

I wasn’t so lucky while I was trying to install this version. Below is my experience for two days. I’ve tried to install invenio-rdm first on CentOS 7, then on Ubuntu 18.04.3 LTS, both failed with the dependency error reported below.

I tried to install using Python 3.6.8 and pip3 both in CentOS 7 and Ubuntu.

Bugs

  • Luckily nothing to report as a bug per-se.

What worked well

  • Installation commands are straightforward.
  • No excessive logs.
  • Minimum number of questions.

What didn’t work well

  • There seems to be a dependency problem in PyPi side. Cannot get installation packages to lock dependencies. The process fails with the following error:

Could not find a version that matches invenio-pidstore<1.1.0,>=1.0.0,>=1.1.0
Tried: 0.1.0, 0.1.1, 0.1.2, 0.1.2, 0.1.3, 0.1.3, 1.0.0, 1.0.0, 1.1.0, 1.1.0

I copied and pasted commands to not to miss anything, but it seems --pre doesn’t get passed to pipenv in invenio-cli process however, this is a pure speculation since I didn’t have time to dig deeper in the code.

Wishes for documentation

  • A before you start or requirements section will be nice.
    • Docker and python-3 requirement became evident during installation, causing time-loss for inexperienced users.
  • Documentation is unclear on which operating systems are viable installation targets. CentOS7 and Ubuntu looks good, but installation behaviors change (e.g. - pip3 install invenio-cli command requires sudo in CentOS7.).

Will try to do my best to retry if anyone can propose solutions.

Best regards,

Hakan

9

VM Testing environment:
OS
Red Hat Enterprise Linux Server release 7.7 (Maipo)
node js-6.17.1-1.el7.x86_64
npm -3.10.10-1.6.17.1.1.el7.x86_64
Same result after upgrading these packages to node js-10.17.0-1 and npm 6.11.3

Bugs

Previously reported? what was the workaround

(invrdmtest) ~$ invenio-cli build --dev --pre --lock

Blockquote Locking [packages] dependencies…
✘ Locking Failed!
[pipenv.exceptions.ResolutionFailure]: req_dir=requirements_dir
[pipenv.exceptions.ResolutionFailure]: File “/opt/invenio/inveniovenv/invrdmtest/lib/python3.6/site-packages/pipenv/utils.py”, line 726, in resolve_deps
[pipenv.exceptions.ResolutionFailure]: req_dir=req_dir,
[pipenv.exceptions.ResolutionFailure]: File “/opt/invenio/inveniovenv/invrdmtest/lib/python3.6/site-packages/pipenv/utils.py”, line 480, in actually_resolve_deps
[pipenv.exceptions.ResolutionFailure]: resolved_tree = resolver.resolve()
[pipenv.exceptions.ResolutionFailure]: File “/opt/invenio/inveniovenv/invrdmtest/lib/python3.6/site-packages/pipenv/utils.py”, line 395, in resolve
[pipenv.exceptions.ResolutionFailure]: raise ResolutionFailure(message=str(e))
[pipenv.exceptions.ResolutionFailure]: pipenv.exceptions.ResolutionFailure: ERROR: ERROR: Could not find a version that matches invenio-pidstore<1.1.0,>=1.0.0,>=1.1.0
[pipenv.exceptions.ResolutionFailure]: Tried: 0.1.0, 0.1.1, 0.1.2, 0.1.2, 0.1.3, 0.1.3, 1.0.0, 1.0.0, 1.1.0, 1.1.0
[pipenv.exceptions.ResolutionFailure]: Skipped pre-versions: 1.0.0a1, 1.0.0a1, 1.0.0a2, 1.0.0a2, 1.0.0a3, 1.0.0a3, 1.0.0a4, 1.0.0a4, 1.0.0a5, 1.0.0a5, 1.0.0a6, 1.0.0a6, 1.0.0a7, 1.0.0a7, 1.0.0a8, 1.0.0a8, 1.0.0a9, 1.0.0a9, 1.0.0b1, 1.0.0b1, 1.0.0b2, 1.0.0b2
[pipenv.exceptions.ResolutionFailure]: Warning: Your dependencies could not be resolved. You likely have a mismatch in your sub-dependencies.
First try clearing your dependency cache with $ pipenv lock --clear, then try the original command again.
Alternatively, you can use $ pipenv install --skip-lock to bypass this mechanism, then run $ pipenv graph to inspect the situation.
Hint: try $ pipenv lock --pre if it is a pre-release dependency.
ERROR: ERROR: Could not find a version that matches invenio-pidstore<1.1.0,>=1.0.0,>=1.1.0
Tried: 0.1.0, 0.1.1, 0.1.2, 0.1.2, 0.1.3, 0.1.3, 1.0.0, 1.0.0, 1.1.0, 1.1.0
Skipped pre-versions: 1.0.0a1, 1.0.0a1, 1.0.0a2, 1.0.0a2, 1.0.0a3, 1.0.0a3, 1.0.0a4, 1.0.0a4, 1.0.0a5, 1.0.0a5, 1.0.0a6, 1.0.0a6, 1.0.0a7, 1.0.0a7, 1.0.0a8, 1.0.0a8, 1.0.0a9, 1.0.0a9, 1.0.0b1, 1.0.0b1, 1.0.0b2, 1.0.0b2
There are incompatible versions in the resolved dependencies.
Blockquote

Not clear to me if this is BUG or a conflict with Pipenv and mkvirtualenv

Blockquote invenio-cli run --dev --bg
Starting/Stopping server for RDM application…
Starting rdmtest_es_1 …
Starting rdmtest_mq_1 …
Starting rdmtest_db_1 …
Starting rdmtest_cache_1 … done
Courtesy Notice: Pipenv found itself running within a virtual environment, so it will automatically use that environment, instead of creating its own for any project. You can set PIPENV_IGNORE_VIRTUALENVS=1 to force pipenv to ignore that environment and create its own instead. You can set PIPENV_VERBOSITY=-1 to suppress this warning.
Courtesy Notice: Pipenv found itself running within a virtual environment, so it will automatically use that environment, instead of creating its own for any project. You can set PIPENV_IGNORE_VIRTUALENVS=1 to force pipenv to ignore that environment and create its own instead. You can set PIPENV_VERBOSITY=-1 to suppress this warning.
Error : the command invenio could not be found within PATH or Pipfile’s [scripts] .
Error : the command celery could not be found within PATH or Pipfile’s [scripts] .
Blockquote

What worked well

Simple commands

What didn’t work well

The definition of the environment prior command execution

Wishes for documentation

There is a wiki and coockiecutter is mentioned there is not context within the document, probably there are two ways to install it.

10

Hey, i’m trying to install this alpha version and i’m getting this error Usage: invenio-cli [OPTIONS] COMMAND [ARGS]…
Try “invenio-cli --help” for help.

Error: no such option: --flavour
how can i fix his error?

11

Hello @mickiyas123 The commands have changed a bit for the CLI. Check out the new release (Alpha 2):

Cheers!
Pablo