Problem with some field values not found in query.json

john's Avatar

john

23 Jan, 2017 12:08 PM

Frederik or whoever can help,

I am using the query_json node to read a JSON file with a number of fields:
  - time
  - id
  - name
  - type
  - message

When I read them all at once using a query like "$.data" I get a table with all the fields and rows, but more than half of the text strings under the message field are missing. Instead of the message text I get "<not found>" in more than half of the rows. The missing values are scattered in no obvious pattern.

Yet the messages really are there in the underlying JSON file. And if a do a query from the same source just for the messages, like "$..message", I DO get all of the messages.

Also, if I create a simpler JSON file with fewer fields, I also get all the messages, but only if I eliminate almost all of the other fields.

The same thing happens with some of the other text fields like name. Time and id fields seem unaffected. If my JSON file has both name and message fields, some rows of both will be missing, but if it has only name or only message (along with time and id), no rows are missing.

It looks like the more complex the JSON file structure, the greater the chance that a subset of values for some fields will be "not found". I've looked hard for anything the missing messages have in common and see nothing obvious. It's almost as if some internal process is timing out and returning <not found> in place of text almost at random. Reloading has no effect; the same random subset of message values are missing.

Have you seen anything like this before? Why would the exact same values for a certain field be sometimes found and sometimes not found depending on what else you are querying?

John

  1. Support Staff 1 Posted by Stefan Gabriels on 25 Jan, 2017 09:12 PM

    Stefan Gabriels's Avatar

    Hello John,

    Is it possible to provide an example JSON file so that I can look into this? Thanks.

    Stefan

  2. Support Staff 2 Posted by john on 25 Jan, 2017 10:51 PM

    john's Avatar

    Hi Stefan,

    Thanks for responding.

    I created a simple example which shows the problem (attached). The JSON file contains 50 FaceBook posts with four fields including message. Of these 50 posts, 39 include message values.

    If you just query for the message field ($..message), you get all 39. But if you query for all the data ($.posts.data), you only see 20 values in the message column; the rest are marked <not found>. See attached screenshot.

    In the process of creating this example, I discovered that even though many of the values are missing in the Data view, they are still there if you switch to the Viewer view. And if you do a lookup on the query with only 20 values showing, the missing 19 values reappear.

    So the values really are there. Some of them are just falsely marked as <not found> in the data view.

    I will leave the json_test.json file on my website for a few days so you can verify. The zip file also includes the actual json file. Please let me know what you find out.

    Thanks!

    John

  3. Support Staff 3 Posted by Stefan Gabriels on 27 Jan, 2017 09:42 PM

    Stefan Gabriels's Avatar

    Hello John,

    The problem is indeed just a displaying problem. The messages are still there and you can still query them. The data viewer mistakenly assumes all records have an equal amount of fields and tries to access them by the column number (instead of what you would expect, the key). In CSV files this normally works because the data format expects it. But JSON files can have any number of fields. So in your example, some rows don't have a name property, the ones displaying 'null', those are the ones that display 'not found' as an unfortunate consequence. Thanks for pointing out this bug, and I will put it in the issues list!

    Stefan

Reply to this discussion

Internal reply

Formatting help / Preview (switch to plain text) No formatting (switch to Markdown)

Attaching KB article:

»

Attached Files

You can attach files up to 10MB

If you don't have an account yet, we need to confirm you're human and not a machine trying to post spam.

Keyboard shortcuts

Generic

? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac

Recent Discussions

16 May, 2024 02:11 AM
16 May, 2024 01:58 AM
14 May, 2024 01:29 AM
13 May, 2024 12:36 AM
12 May, 2024 07:46 PM

 

05 May, 2024 04:38 PM
03 May, 2024 06:50 AM
03 May, 2024 12:30 AM
03 May, 2024 12:09 AM
19 Apr, 2024 04:41 AM
16 Apr, 2024 06:18 PM