InvenioRDM Alpha 2 (November Release)

Projects Invenio RDM
1

First of all thank you once again for your feedback on the October release! Let’s move one more step forward :smiley:

What do you need to do?

  1. Leave all feedback as comments to this thread using the template below.
  2. Report any bugs you experience (simply reply with comments to this thread).
  3. Report a) what worked well and b) what did not work well (comments can be related to anything not only the software).
  4. Report what you would expect to see documented in the documentation and what you would like to know more about. We are working on this aspect according to the previous release feedback :grinning:.

You can use the following template:

## Bugs

## What worked well

## What didn't work well

## Wishes for documentation

What will you be doing?

You will follow a set of simple commands that will allow you to deploy an InvenioRDM instance in a containerized environment.

What shall be the outcome?

You should have an instance, running fully in docker containers, with ten records that you can query through the API. You should also be able to customize the logo and the permissions.

What should I be testing?

You should test the deployment process, meaning that the commands work and produce the expected output. Note that there are some small known issues, these are noted at the end of each section. Finally, know that the datamodel is just a preliminary one; there will be some time to discuss it in January at the collaborators meeting: make sure to sign up so you can give your input!

Hands-on! Follow the Wiki documentation along with the examples and explanations.

Troubleshooting

We will update this as we get information during next week.

Stop all existing running docker containers (otherwise, you may have port conflicts):

docker stop \$(docker ps -a -q)
1 Like
2

Bugs

Setup issue with files location creation

When running invenio-cli setup --prod, the following error occurs:

$ invenio-cli setup --prod
Setting up environment for RDM application...
Setting up server...
Starting containers...
Creating database...
Creating indexes...
Creating files location...
Could not update config. Env dir does not extis.

I had a brief look for the cause and created an issue for this in invenio-cli: #inveniosoftware/invenio-cli/issues/61 (I can’t post more than two links apparently, because I’m a new member in the forum… :confused:)

There’s a also a minor typo, s/extis/exist.

I guess for others wanting to continue (until this is fixed), getting around requires doing something like:

$ docker-compose -f docker-compose.full.yml exec web-api bash
[root@a45018e1503f src]# invenio files location --default 'default-location' /opt/invenio/var/instance/

# Also run the rest of the commands
[root@a45018e1503f src]# invenio roles create admin
[root@a45018e1503f src]# invenio access allow superuser-access role admin

invenio.cfg “automagic” variables

At some point in the “Configuration” section, the following is mentioned:

# automatically set configuration variables
ACCOUNTS_SESSION_REDIS_URL=...
# ...

Besides being actually valid Python code (because of the Ellipsis literal) but obviously an invalid configuration value, it might cause some other confusion:

When I read the comment a silly thought came through my mind that setting something to ... would somehow “magically” make it get replaced by the default value (e.g. from the specific extension). I’m pretty besides being a “Difficult problem ™” overall, it’s also probably not that useful/intuitive.

Maybe removing the specific lines (and just keeping the Permission Policy class only) or including an actual commented-out config key-value pair as an example, would clarify things.

What worked well

  • The quick and compact instructions and feature exploration is fantastic!

What didn’t work well

  • Some scripts (e.g. build) take some time to execute and the user ends up with a static terminal without knowing if there’s any progress. Not sure how easy it would be to add a progress bar/spinner though…
  • When updating the invenio.cfg file (e.g. for changing permissions), having to run invenio-cli build ... feels a bit heavy. Doing a container.put_archive call would help a lot (not sure what the drawbacks/issues are though). Another thing to explore is using flask run --extra-files ...

Wishes for documentation

  • For the “Install the CLI” section, recommending pipenv to install the package seems a bit counter-intuitive. The way I perceive it, invenio-cli is a general utility (similar to vue-cli and create-react-app), so basically something one would pip install --user --upgrade invenio-cli (or manage with pipx).
  • Some info regarding version control of the generated files (e.g. running git init at some point)
  • Adding a .gitignore for e.g. logs and other common files might be necessary then
  • For the authenticated REST API calls, using the CLI for creating the admin user account and tokens via invenio tokens create ... could also be a nice alternative option
  • A mini-changelog for each release would help with knowing what to look further into regarding new features
3

Hi @aioannid !

Thanks a lot for the feedback. The issue has been addressed and fixed, thank you very much for reporting. Using the invenio-cli version v1.0.0a6 should not produce the error.

Cheers,
Pablo

4

Thanks a lot, it works now! Tested by upgrading to 1.0.0a6 and running invenio-cli setup --prod --force, and it goes through all of the steps successfully :slight_smile:

5

Bugs

What worked well

This release is awesome! Installed easily and was able to change logo and customize permissions. The file organization changes make sense and I like how all the configuration is in one directory.

What didn’t work well

What’s the default admin account? It’s not clear how to give permissions to a created user account.

I had the same ACCOUNTS_SESSION_REDIS_URL=… question as Alex. What is this for? It seemed fine when I commented it out.

Wishes for documentation

The section “Build the docker images” mentions the --lock flag, but it isn’t in the sample command. It seems like locking happens by default.

In the “Have fun” section there is an example called “You can also manually index a record:”. I’d describe this as “You can create a new record using the API.” Index seems like too technical a term.

6

Thanks for those notes about the wiki. I’ve updated it with them.

I thought ACCOUNTS_SESSION_REDIS_URL=… was going to help people situate themselves in the file to make the edits… but that completely missed the obvious confusion that would create :sweat_smile:

7

Bugs

i did not see any bugs. :slight_smile:

What worked well?

for me everything worked very smooth.

What didn’t worked?

changing the logo.
i can not use the environment variable of ‘THEME_LOGO’ to set my logo anymore.
and i am only restricted to use the SVG’s to add the logo. which means i have to delete the old one and add the same name logo.svg to replace the logo.

Wishes for documentation

how to get access to the codes like html, python files and etc.

8

@mb-wali Thanks a lot for your feedback, I have identified the issue with the logo and will fix it asap. We will fix it asap :slight_smile:

9

@mb-wali for the “Wishes for documentation” part. Did you mean that

You would like to have documentation explaining how to edit the code (python or html or css) of your personal instance to add your own functionality / customize (beyond just configuration).

or

Where are the different modules hosted on GitHub and who is in charge of them?

10

@fenekku thanks for the reply.
actually both. :slight_smile:

11

Hey, i followed this command to install invenio alpha2 https://github.com/inveniosoftware/invenio-app-rdm/wiki when i run invenio-cli init --flavour=RDM i get this error
Traceback (most recent call last):
File “/home/dell/.local/share/virtualenvs/dell-c9i2WUPg/bin/invenio-cli”, line 8, in
sys.exit(cli())
File “/home/dell/.local/share/virtualenvs/dell-c9i2WUPg/local/lib/python2.7/site-packages/click/core.py”, line 764, in call
return self.main(*args, **kwargs)
File “/home/dell/.local/share/virtualenvs/dell-c9i2WUPg/local/lib/python2.7/site-packages/flask/cli.py”, line 586, in main
return super(FlaskGroup, self).main(*args, **kwargs)
File “/home/dell/.local/share/virtualenvs/dell-c9i2WUPg/local/lib/python2.Traceback (most recent call last):
File “/home/dell/.local/share/virtualenvs/dell-c9i2WUPg/bin/invenio-cli”, line 8, in
sys.exit(cli())
File “/home/dell/.local/share/virtualenvs/dell-c9i2WUPg/local/lib/python2.7/site-packages/click/core.py”, line 764, in call
return self.main(*args, **kwargs)
File “/home/dell/.local/share/virtualenvs/dell-c9i2WUPg/local/lib/python2.7/site-packages/flask/cli.py”, line 586, in main
return super(FlaskGroup, self).main(*args, **kwargs)
File “/home/dell/.local/share/virtualenvs/dell-c9i2WUPg/local/lib/python2.7/site-packages/click/core.py”, line 717, in main
rv = self.invoke(ctx)
File “/home/dell/.local/share/virtualenvs/dell-c9i2WUPg/local/lib/python2.7/site-packages/click/core.py”, line 1132, in invoke
cmd_name, cmd, args = self.resolve_command(ctx, args)
File “/home/dell/.local/share/virtualenvs/dell-c9i2WUPg/local/lib/python2.7/site-packages/click/core.py”, line 1171, in resolve_command
cmd = self.get_command(ctx, cmd_name)
File “/home/dell/.local/share/virtualenvs/dell-c9i2WUPg/local/lib/python2.7/site-packages/flask/cli.py”, line 527, in get_command
self._load_plugin_commands()
File “/home/dell/.local/share/virtualenvs/dell-c9i2WUPg/local/lib/python2.7/site-packages/flask/cli.py”, line 523, in _load_plugin_commands
self.add_command(ep.load(), ep.name)
File “/home/dell/.local/share/virtualenvs/dell-c9i2WUPg/local/lib/python2.7/site-packages/pkg_resources/init.py”, line 2443, in load
return self.resolve()
File “/home/dell/.local/share/virtualenvs/dell-c9i2WUPg/local/lib/python2.7/site-packages/pkg_resources/init.py”, line 2449, in resolve
module = import(self.module_name, fromlist=[‘name’], level=0)
File “/home/dell/.local/share/virtualenvs/dell-c9i2WUPg/local/lib/python2.7/site-packages/invenio_cli/cli.py”, line 15, in
from pathlib import Path
ImportError: No module named pathlib7/site-packages/click/core.py”, line 717, in main
rv = self.invoke(ctx)
File “/home/dell/.local/share/virtualenvs/dell-c9i2WUPg/local/lib/python2.7/site-packages/click/core.py”, line 1132, in invoke
cmd_name, cmd, args = self.resolve_command(ctx, args)
File “/home/dell/.local/share/virtualenvs/dell-c9i2WUPg/local/lib/python2.7/site-packages/click/core.py”, line 1171, in resolve_command
cmd = self.get_command(ctx, cmd_name)
File “/home/dell/.local/share/virtualenvs/dell-c9i2WUPg/local/lib/python2.7/site-packages/flask/cli.py”, line 527, in get_command
self._load_plugin_commands()
File “/home/dell/.local/share/virtualenvs/dell-c9i2WUPg/local/lib/python2.7/site-packages/flask/cli.py”, line 523, in _load_plugin_commands
self.add_command(ep.load(), ep.name)
File “/home/dell/.local/share/virtualenvs/dell-c9i2WUPg/local/lib/python2.7/site-packages/pkg_resources/init.py”, line 2443, in load
return self.resolve()
File “/home/dell/.local/share/virtualenvs/dell-c9i2WUPg/local/lib/python2.7/site-packages/pkg_resources/init.py”, line 2449, in resolve
module = import(self.module_name, fromlist=[‘name’], level=0)
File “/home/dell/.local/share/virtualenvs/dell-c9i2WUPg/local/lib/python2.7/site-packages/invenio_cli/cli.py”, line 15, in
from pathlib import Path
ImportError: No module named pathlib
i tried installing sudo apt-install pathlib but it didn’t work
i really appreciate what you guys are doing

12

Hello @mickiyas123 I am sorry for the inconvenience. I see we did not clearly specify that the RDM only supports Python 3 (max 3.6). In python 2.7 which I can see is your use case the pathlib was named differently IIRC. In any case, the issue should be solved by installing it on top of Python 3.6.

Note that Python 2.7 is reaching end of life next month, and that’s why we are not supporting it.

Once again, sorry for the inconvenience.

Cheers,
Pablo

13

Ah I knew I shouldn’t have asked an either-or question :smile: !

Documentation to add/edit code and so on, will trickle in as we work on that aspect of the project. Already, one can get very far with just configuration given that invenio.cfg is a simple Python module. Any configuration variable that could be given custom code can already be dealt with by putting the code in invenio.cfg. The amount of freedom we want to allow beyond that is debatable. In fact I’ve outlined the “customization” hierarchy issue here: https://github.com/inveniosoftware/invenio-app-rdm/issues/1 . Do you think it makes sense? Share your thoughts on it if we missed things.

As to where modules are hosted, the search bar on the inveniosoftware GitHub organization is your friend. Modules specific to InvenioRDM will have rdm in their name (see naming convention: https://github.com/inveniosoftware/rfcs/blob/master/framework/0008-module-naming-conventions.md).

As for who is working on what, one thing that could be nice, is being more specific in the people section of the inveniosoftware site (e.g., Bob works on InvenioRDM, Alice works on CDS). On the flip side, given CERN’s turnover / the collaborative nature of the project, using the common forums (this discourse, the gitter chat and participating in community workshops) makes sure you are more likely to get at least one person to respond to your questions. What would you like to see to address this? And actually what is the problem this would help you solve? I assumed it was to target your questions to individuals but maybe you meant something else :slight_smile:!

14

Installation on CentOS 7 went better this time, without too many problems. Since I’m doing my tests in a VM, iterations are not painful however, not accepting connections from outside limits testing the UI side.

Bugs

  • On CentOS 7, the containers do not stop completely after invenio-cli build --pre --prod command. This results exceptions and a crippled setup when invenio-cli setup --prod command is run.
    • Installation can be fixed by stopping all containers by hand and running invenio-cli setup --prod --force.

What worked well

General setup process runs well. It does what it advertises (for the most part, see bugs).

What didn’t work well

  • Redirection of all logs is a huge plus however, exceptions and other errors are also masked. This resulted in delayed diagnosis on the container problem (see Bugs).
  • docker ps command reports elasticsearch container as unhealthy.

Wishes for documentation

On CentOS 7, following differences are spotted.

  • pipenv is not packaged in CentOS 7. User needs to install it with pip3 install --user pipenv command.
  • Since everything is inside the virtualenv now, user needs to switch into the virtualenv to continue installation by running pipenv shell command.

Will try to add another reply after exploring the customization part.

Cheers,

Hakan

16

Thanks for the feedback @hbayindir !

Although there are some improvements in invenio-cli for this release (upcoming today!), I don’t think the issues you are raising will be completely dealt with. It will get more love and changes next release.

pipenv is indeed not packaged with CentOS or any other OS. I will adjust the wiki text for that today, so it is more clear that pipenv run or pipenv shell needs to be used throughout.

The ES container issue is something that showed up with ES 7. I’ve found that version of ES to be slower to start but be more performant after. Our plans for invenio-cli will make this less of a common problem in your day to day workflow.

Thanks!