metadata XML upload #394

tomkralidis · 2012-09-17T16:55:55Z

This PR allows for users to upload an XML metadata document as part of their data upload. Notes:

on new data upload, if user chooses XML field, title and abstract are disabled and values will be picked up by XML metadata parser/setter
layers uploaded with XML metadata can update their metadata by uploading a new XML document
the XML metadata document (on create or update) is processed and set with GeoNode specific values (URLs, etc.) before being set in the DB
supports Dublin Core, FGDC CSDGM and ISO 19139:2007

This will likely bring up issues of signals and what fields should go where again, hence I've not included tests until we agree (where tests will go will depend on this discussion).

…ident

ischneider · 2012-09-17T23:01:26Z

geonode/layers/utils.py

+
+    if 'xml' in files:
+        md_xml, md_title, md_abstract = update_metadata(layer_uuid, open(files['xml']).read(), saved_layer)
+        Layer.objects.filter(uuid=layer_uuid).update(


This model update looks a lot like the 4 lines following (492-495). Is there something I'm missing?

ingenieroariel · 2012-09-19T16:04:45Z

The use case we have in the bank for this is the following:

We receive the metadata from consultants in ISO format, but would like to use GeoNode to keep it up to date. Using GeoNode as a metadata editor, since the people editing it are not going to be the ones who created it in the first place.

As such, the main use for this functionality would be to upload data in xml format (for the initial values) and edit after that. This pull request blocks the ability for people to edit metadata even if they had an xml file that only contained title and nothing else, as such I am -1 on merging it until editing metadata is supported.

jj0hns0n · 2012-09-19T16:15:10Z

To be clear, @ingenieroariel, you want the user to be able to upload xml
metadata, then edit it via the metadata editor? @tomkralidis, I know we
have discussed several pitfalls and problems with this use case, can you
elaborate on those?

On Wed, Sep 19, 2012 at 9:04 AM, Ariel Núñez [email protected]:

The use case we have in the bank for this is the following:

We receive the metadata from consultants in ISO format, but would like to
use GeoNode to keep it up to date. Using GeoNode as a metadata editor,
since the people editing it are not going to be the ones who created it in
the first place.

As such, the main use for this functionality would be to upload data in
xml format (for the initial values) and edit after that. This pull request
blocks the ability for people to edit metadata even if they had an xml file
that only contained title and nothing else, as such I am -1 on merging it
until editing metadata is supported.

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/394#issuecomment-8696125.

tomkralidis · 2012-09-19T16:49:19Z

@ingenieroariel the main drawback is that if someone uploads a rich XML document, given that GeoNode's Layer model is a limited subset, parsing the XML to GeoNode model.will result in a lossy transfer.

The blocking is by design so that edits are done via XML if XML was the initial upload mechanism (if I had an XML document with a single element, I wouldn't upload it at all and just use the editor).

This has been discussed previously, and is a known limitation.

Options:

If we want full XML document <-> GeoNode model compatibility, this requires some heavy lifting on the model, as well as the front end UI (which was not in scope of this PR)
if we are okay with having XML uploaded and then discarded to make way for quickly filling in the form by way of XML upload, then we can remove the hooks that force 'edit by XML upload only', and always save XML to the DB per ISO

jj0hns0n · 2012-09-19T16:52:10Z

I think we need to go with this second option here in the short term and
then look into the first option in more detail at some later date.
Basically chalk up this lossiness as a known limitation.

On Wed, Sep 19, 2012 at 9:49 AM, Tom Kralidis [email protected]:

@ingenieroariel https://github.com/ingenieroariel the main drawback is
that if someone uploads a rich XML document, given that GeoNode's Layer
model is a limited subset, parsing the XML to GeoNode model.will result in
a lossy transfer.

The blocking is by design so that edits are done via XML if XML was the
initial upload mechanism (if I had an XML document with a single element, I
wouldn't upload it at all and just use the editor).

This has been discussed previously, and is a known limitation.

Options:

If we want full XML document <-> GeoNode model compatibility, this
requires some heavy lifting on the model, as well as the front end UI
(which was not in scope of this PR)

if we are okay with having XML uploaded and then discarded to make
way for quickly filling in the form by way of XML upload, then we can
remove the hooks that force 'edit by XML upload only', and always save XML
to the DB per ISO

—
Reply to this email directly or view it on GitHubhttps://github.com/metadata XML upload #394#issuecomment-8697660.

ingenieroariel · 2012-09-19T16:54:50Z

I agree with Jeffrey, let's go with option 2.

On Wed, Sep 19, 2012 at 12:52 PM, Jeffrey Johnson
[email protected]:

I think we need to go with this second option here in the short term and
then look into the first option in more detail at some later date.
Basically chalk up this lossiness as a known limitation.

On Wed, Sep 19, 2012 at 9:49 AM, Tom Kralidis [email protected]:

@ingenieroariel https://github.com/ingenieroariel the main drawback
is
that if someone uploads a rich XML document, given that GeoNode's Layer
model is a limited subset, parsing the XML to GeoNode model.will result
in
a lossy transfer.

The blocking is by design so that edits are done via XML if XML was the
initial upload mechanism (if I had an XML document with a single
element, I
wouldn't upload it at all and just use the editor).

This has been discussed previously, and is a known limitation.

Options:

If we want full XML document <-> GeoNode model compatibility, this
requires some heavy lifting on the model, as well as the front end UI
(which was not in scope of this PR)

if we are okay with having XML uploaded and then discarded to make
way for quickly filling in the form by way of XML upload, then we can
remove the hooks that force 'edit by XML upload only', and always save
XML
to the DB per ISO

—
Reply to this email directly or view it on GitHub<
https://github.com/GeoNode/geonode/pull/394#issuecomment-8697660>.

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/394#issuecomment-8697773.

jj0hns0n · 2012-09-19T16:57:33Z

I think its even ok to include this kind of 'warning' right in the form
itself. I.e. if metadata was originally uploaded, and then the user tries
to edit it with the editor, just show a warning in the page that notifies
them that some of their original metadata could be lost. Not sure if we
can actually determine if it will be lost, but if so, then just show it
then.

On Wed, Sep 19, 2012 at 9:54 AM, Ariel Núñez [email protected]:

I agree with Jeffrey, let's go with option 2.

On Wed, Sep 19, 2012 at 12:52 PM, Jeffrey Johnson
[email protected]:

I think we need to go with this second option here in the short term and
then look into the first option in more detail at some later date.
Basically chalk up this lossiness as a known limitation.

On Wed, Sep 19, 2012 at 9:49 AM, Tom Kralidis [email protected]:

@ingenieroariel https://github.com/ingenieroariel the main drawback
is
that if someone uploads a rich XML document, given that GeoNode's
Layer
model is a limited subset, parsing the XML to GeoNode model.will
result
in
a lossy transfer.

The blocking is by design so that edits are done via XML if XML was
the
initial upload mechanism (if I had an XML document with a single
element, I
wouldn't upload it at all and just use the editor).

This has been discussed previously, and is a known limitation.

Options:

If we want full XML document <-> GeoNode model compatibility, this
requires some heavy lifting on the model, as well as the front end UI
(which was not in scope of this PR)

if we are okay with having XML uploaded and then discarded to make
way for quickly filling in the form by way of XML upload, then we can
remove the hooks that force 'edit by XML upload only', and always save
XML
to the DB per ISO

—
Reply to this email directly or view it on GitHub<
https://github.com/GeoNode/geonode/pull/394#issuecomment-8697660>.

—
Reply to this email directly or view it on GitHub<
https://github.com/GeoNode/geonode/pull/394#issuecomment-8697773>.

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/394#issuecomment-8697859.

tomkralidis · 2012-09-21T16:03:04Z

@ingenieroariel @jj0hns0n : updates made per recommendation. For review and comment.

metadata XML upload

tomkralidis added 13 commits August 31, 2012 19:47

update form defs

d4cc7f8

add template

2443a76

update form defs

10e6924

add fields to support metadata upload

a8595dc

Merge branch 'dev' of github.com:GeoNode/geonode into md-xml-upload

c47a13c

add metadata upload handlers

a63a396

sync to dev

df54d00

remove merge artifact

47ce074

fix merge conflicts origin/dev

bd9c9b7

clean up before PR

c92f287

Merge branch 'dev' of github.com:GeoNode/geonode into md-xml-upload

72594eb

add migration and reset geoserver configs which were committed by acc…

248cd58

…ident

sync geoserver config XML files again

1aa55b0

ischneider reviewed Sep 17, 2012
View reviewed changes

tomkralidis added 4 commits September 19, 2012 01:54

implement fixes based on @ischneider comments (#394)

5b2a84f

remove unused modules

1597304

Merge branch 'dev' of github.com:GeoNode/geonode into md-xml-upload

f47dbe5

always generate anytext

1e0da81

tomkralidis added 2 commits September 20, 2012 12:35

first pass remove XML only layer editing

cfbcaf4

first pass remove XML only layer editing

4f1d9ac

tomkralidis added 5 commits September 20, 2012 12:35

Merge branch 'dev' of github.com:GeoNode/geonode into md-xml-upload

82d752a

remove unused import

ffb59c4

remove unused import

9fc6f1a

Merge branch 'dev' of github.com:GeoNode/geonode into md-xml-upload

bafd08b

if metadata was originally uploaded, show a warning message of lossiness

1a590b1

Merge branch 'dev' of github.com:GeoNode/geonode into md-xml-upload

e9b41e7

ingenieroariel added a commit that referenced this pull request Sep 22, 2012

Merge pull request #394 from tomkralidis/md-xml-upload

f498818

metadata XML upload

ingenieroariel merged commit f498818 into GeoNode:dev Sep 22, 2012

cspanring pushed a commit to MAPC/geonode that referenced this pull request Feb 6, 2013

implement fixes based on @ischneider comments (GeoNode#394)

6412d0a

cspanring pushed a commit to MAPC/geonode that referenced this pull request Feb 6, 2013

Merge pull request GeoNode#394 from tomkralidis/md-xml-upload

b56cae2

metadata XML upload

marthamareal pushed a commit to marthamareal/geonode that referenced this pull request Sep 24, 2021

fix async loading of plugins (GeoNode#394)

d35439f

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

metadata XML upload #394

metadata XML upload #394

Uh oh!

tomkralidis commented Sep 17, 2012

Uh oh!

ischneider Sep 17, 2012

Uh oh!

ingenieroariel commented Sep 19, 2012

Uh oh!

jj0hns0n commented Sep 19, 2012

Uh oh!

tomkralidis commented Sep 19, 2012

Uh oh!

jj0hns0n commented Sep 19, 2012

Uh oh!

ingenieroariel commented Sep 19, 2012

Uh oh!

jj0hns0n commented Sep 19, 2012

Uh oh!

tomkralidis commented Sep 21, 2012

Uh oh!

Uh oh!

metadata XML upload #394

metadata XML upload #394

Uh oh!

Conversation

tomkralidis commented Sep 17, 2012

Uh oh!

ischneider Sep 17, 2012

Choose a reason for hiding this comment

Uh oh!

ingenieroariel commented Sep 19, 2012

Uh oh!

jj0hns0n commented Sep 19, 2012

Uh oh!

tomkralidis commented Sep 19, 2012

Uh oh!

jj0hns0n commented Sep 19, 2012

Uh oh!

ingenieroariel commented Sep 19, 2012

Uh oh!

jj0hns0n commented Sep 19, 2012

Uh oh!

tomkralidis commented Sep 21, 2012

Uh oh!

Uh oh!