InvenioRDM Alpha 9 (July Release)

We are happy to announce InvenioRDM Alpha 9 (July release)!

What’s new?

The July release adds translation support and starts integrating the major backend development of last release into InvenioRDM. In particular, draft functionality has been added at the API level and is used under the hood for deposits. To achieve this integration, the hard problems like responsibility separation, error handling, pagination, linking and internal PID management were solved. Other improvements and more details follow.

Update invenio-cli to version 0.15.0 and follow the documentation to get started!

Module translation

Thanks to the efforts of TÜBITAK, InvenioRDM was setup for internationalization: all text has been registered for translation, the Transifex service was enabled and even a VSCode extension is in the works to help future translators. Turkish was added as a first alternative language! How to go about translating and using the various tools will be laid out next month once we have more experience under our belt :wink: !

Library improvements

Error handling, default MIME type headers and uniform body + querystring deserialization have been added to Flask-Resources, our Invenio-agnostic Flask REST library.

Note that invenio-resources is now invenio-records-resources (it includes the former invenio-records-agent) and invenio-drafts-resources is the new package that houses the draft functionality.

Behind the scenes, persistent identifiers have also seen some work.

New API integration (including draft functionality)

The deposit page now uses the new API to create records by creating drafts and publishing them immediately. In the next release, we hope you will be able to save and publish separately. That being said, you can already do so on the command-line! Make sure to follow the updated documentation to see creation, publication, retrieval, and search in action there.

In addition to these tent-pole features, Python 3.7 support and general containerization was further improved thanks to Cottage Labs through an updated base image.

Community domains vocabulary

The list of available domains/disciplines that can be added to a community is now based on a pre-defined (and configurable) vocabulary.

What do you need to do?

You can install the latest invenio-cli v0.15.0, create a new instance and see how it looks like!

Install (TL;DR)

If you previously installed InvenioRDM, make sure you have the latest Docker image of your choice according to the Python version:

docker pull inveniosoftware/centos7-python:3.6
docker pull inveniosoftware/centos8-python:3.7
docker pull inveniosoftware/centos8-python:3.8

To install:

pip install --upgrade invenio-cli 
invenio-cli init rdm
cd my-site
invenio-cli containerize --pre
invenio-cli demo --containers

To destroy the Python virtualenv, and remove the docker containers run:

cd my-site
pipenv --rm
docker-compose -f docker-compose.full.yml down

Feedback

As always, we welcome your feedback. When you reply to this post your message should be pre-populated with the classic template (bugs, what worked well, what didn’t work well, wishes for documentation).

Here is the template to give feedback if it’s not automated:

## Bugs

## What worked well

## What didn't work well

## Wishes for documentation

Take care and stay safe!

Bugs

While publishing a record using the deposits UI, there are not specific error messages from required information. For example if the license type is missing and the resource type, the error message is the following one: The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application.
however in the uwsgi.logs you can see marshmallow.exceptions.ValidationError: {'access_right': ['Missing data for required field.'], 'resource_type': ['Missing data for required field.']}

-Creators (name) does not have the red star to emphasize that is a required field

-the search field appears to be enabled to search only on deposits or to search on communities the return page should clearly define where the search is performed

-I tried to query the curl -k -XGET ‘https://testhost/api/rdm-records
{“message”:“The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application.”,“status”:500}

Looking at uwsgi I see this [2020-08-04 21:37:28,026] ERROR in app: Exception on /rdm-records [GET]
File "/opt/invenio/invenio-venv/latest/lib/python3.6/site-packages/invenio_pidstore/fetchers.py", line 66, in recid_fetcher pid_value=str(data[pid_field]), KeyError: 'recid'

What worked well

no issues during installation

Thanks Carlos for this feedback!

The error display on the frontend is indeed not fully integrated yet (it also had issues prior). We have an issue for it: Ensure all fields are displaying correctly their error state · Issue #22 · inveniosoftware/react-invenio-deposit · GitHub where I also placed your comments.

-Creators (name) does not have the red star to emphasize that is a required field

This is a tricky one: we haven’t quite settled on how to deal with unknown creators. A number of the sites have historical records where the creator is unknown. A good recommendation by Tom Morrell is to make use of DataCite’s Unknown author justification metadata (see Require to explain choice of Unknown in deposit page · Issue #89 · inveniosoftware/invenio-rdm-records · GitHub and Fix json schema occurrences for creators · Issue #86 · inveniosoftware/invenio-rdm-records · GitHub ), but we haven’t gotten to tackling this yet.

Here do you mean that a universal search would be nice? Make it possible to search on records and communities for instance in the same search?

Did you get this on a new instance with new drafts or records? or were there old records in there? We had a bug where old records (and for a while drafts) weren’t indexed with their recid so this problem would occur…

(Since I can’t edit the original post)

If you have an old installation you used to just look around (and you are fine destroying it), these more detailed instructions may help you (courtesy of @mcarson):

# Destroy the Python virtualenv for a previous installation and remove the docker containers:
cd my-previous-installation
pipenv --rm
docker-compose -f docker-compose.full.yml down

# Install new release
docker pull inveniosoftware/centos8-python:3.7
pip install --upgrade invenio-cli 
invenio-cli init rdm
cd ../my-site
invenio-cli containerize --pre
invenio-cli demo --containers

Just been trying it out, a couple of minor things, which you are probably aware of:

  • invenio-cli demo --local doesn’t work - complains about docker not being installed (I don’t have dockerized services, fine if this isn’t supported…).

  • While adding a record to a community, if you press space in the message box it disappears (ie, only 1 word requests are accepted - which might be by design? :slight_smile: ).

Thanks for checking it out Dan!

invenio-cli demo --local doesn’t work - complains about docker not being installed (I don’t have dockerized services, fine if this isn’t supported…).

Yes, invenio-cli demo --local currently expects the services to have been dockerized as per the step before, so it won’t work if setup differently.

community

I will hand that one off to @aioannid and the communities team :slight_smile:

Blockquote
Did you get this on a new instance with new drafts or records? or were there old records in there? We had a bug where old records (and for a while drafts) weren’t indexed with their recid so this problem would occur…`

The only information preserved from the previous instance was the Invenio.cfg file.

Ok @fenekku I run a clean installation (not using custom metadata defined in the invenio.cfg)
I was able to publish the record and query it.

Blockquote
Here do you mean that a universal search would be nice? Make it possible to search on records and communities for instance in the same search?

I think universal search is useful, but my point is not that. I think that visually the user needs to now that the return is either from communities or deposits.

I agree that it should be a clear at all times if you are browsing/searching in the context of communities or records.

Where it says ‘communities’ and ‘upload’ in the header, I’d prefer to see something like toggle buttons which you can both click and see which mode (communities/records) is selected. Here’s a simple example of something we’ve done like this - Faizulloev . I find this sufficient, but for a general use repository you probably want to make it even clearer?

Related to this, I’d also push the ‘uploads’ button over into the right-menu as its function is related to the user’s content rather than the whole repository. So the header would contain 3 groups of elements

  • left - branding : Logo
  • centred - search : Search Box / Search Context selector
  • right - user : (my) uploads / drop-down with logout/profile etc

Hmm if I get this right, you mean that on the communities page for instance, there is a need to distinguish between the search bar for records and the search bar for communities (or even the search bar for records in a specific community) better. Is that right? And Dan makes the point of moving communities / uploads around as a way to help this in part too.

I know there is work to be done to handle different search bars on the same page correctly: https://github.com/inveniosoftware/react-searchkit/issues/117 (it’s a little low-level, but means that to be able to customize each search app instance, there are some aspects that need to be resolved).

The header bar layout can be customized so this is possible to do. That being said, having the default layout be as good as it can be would definitely be best :slight_smile: . @lnielsen can comment on that.

Having reflected on this a little, I think there’s a developer point of view to my approach - communities are one set of data, records/deposits are another. But users won’t interact with them in the same way. I wonder now if the better way is to have the header always search records, and have a different search box when you’re looking for a particular community?

Either way @cfcamboa’s point stands - users should know what they’re searching amongst when they click ‘search’.

I just installed the cli according to documentation. When trying to containerize and launch my preview instance (via: invenio-cli containerize --pre) i get the following error message:

[Warning] One or more build-args [ENVIRONMENT] were not consumed
Successfully built aed192529455
Successfully tagged my-site:latest
Creating my-site_mq_1    ... done
Creating my-site_es_1    ... done
Creating my-site_cache_1 ... done
Creating my-site_db_1    ... done
Creating my-site_worker_1  ... done
Creating my-site_web-api_1 ... done
Creating my-site_web-ui_1  ... done
Creating my-site_frontend_1 ... done
Creating database...
ERROR:root:[2020-08-27 13:35:00,846] ERROR in app: Failed to initialize entry point: invenio_app_rdm = invenio_app_rdm.theme.views:blueprint
Traceback (most recent call last):
  File "/opt/rh/rh-python36/root/usr/bin/invenio", line 8, in <module>
    sys.exit(cli())
  File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/flask/cli.py", line 586, in main
    return super(FlaskGroup, self).main(*args, **kwargs)
  File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/click/core.py", line 1254, in invoke
    cmd_name, cmd, args = self.resolve_command(ctx, args)
  File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/click/core.py", line 1297, in resolve_command
    cmd = self.get_command(ctx, cmd_name)
  File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/flask/cli.py", line 542, in get_command
    rv = info.load_app().cli.get_command(ctx, name)
  File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/flask/cli.py", line 381, in load_app
    app = call_factory(self, self.create_app)
  File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/flask/cli.py", line 117, in call_factory
    return app_factory(script_info)
  File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/invenio_base/app.py", line 154, in create_cli_app
    app = create_app(debug=get_debug_flag())
  File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/invenio_base/app.py", line 120, in _create_app
    modules=blueprints,
  File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/invenio_base/app.py", line 196, in blueprint_loader
    _loader(app, loader_init_func, entry_points=entry_points, modules=modules)
  File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/invenio_base/app.py", line 233, in _loader
    init_func(ep.load())
  File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2462, in load
    return self.resolve()
  File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2468, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/invenio_app_rdm/theme/views.py", line 21, in <module>
    from invenio_rdm_records.resources import BibliographicDraftActionResource, \
  File "/opt/rh/rh-python36/root/usr/lib/python3.6/site-packages/invenio_rdm_records/resources.py", line 13, in <module>
    from invenio_drafts_resources.serializers import DraftJSONSerializer
ImportError: cannot import name 'DraftJSONSerializer'
Creating files location...

Any help or hints will be appreciated. Thank you

Thanks @Faha for trying InvenioRDM!

This is due to a bug with dependency resolution in the July release: there was no upper bound for some modules, so newer August modules are picked up (and they are obviously incompatible with other modules of that release).

This is resolved in the August release; it should not happen for future releases (at least for the modules our team is responsible for :wink: ) either. Luckily, the August release is just around the corner! It should land early next week and then you can try it out again.

cheers,