1
$\begingroup$

The Wolfram Language has a Query function that can traverse data structures and apply functions at different levels of the structure. I am working with multi-level JSON structures and need a function that has similar functionality as that of Query in the Wolfram Language.

Which Python package and function(s) best replicates this?

For a minimal working example, say I have the following JSON structure. (String escapes omitted for simplicity)

x = {
    "Dims1":[
        {
            "Apple":{
                "Baking":[
                    "Pie",
                    "Tart"
                ],
                "Plant":"Tree",
                "Tons":{
                    "2017":1.23e1,
                    "2018":1.12e1
                }
            }
        },
        {
            "Tomato":{
                "Cooking":[
                    "Stew",
                    "Sauce"
                ],
                "Plant":"Vine",
                "Tons":{
                    "2017":8.1,
                    "2018":8.3
                }
            }
        },
        {
            "Banana":{
                "Name":"Banana",
                "Baking":[
                    "Bread"
                ],
                "Cooking":[
                    "Fried"
                ],
                "Plant":"Arborescent",
                "Tons":{
                    "2017":0.8,
                    "2018":0.5
                }
            }
        }
    ],
    "Dims2":[
        {
            "Apple":{
                "Name":"Apple",
                "Baking":[
                    "Pie",
                    "Tart"
                ],
                "Plant":"Tree",
                "Tons":{
                    "2017":1.31e1,
                    "2018":1.01e1
                }
            }
        },
        {
            "Sweet Potato":{
                "Cooking":[
                    "Fried",
                    "Steamed"
                ],
                "Baking":[
                    "Pie"
                ],
                "Plant":"Vine",
                "Tons":{
                    "2017":1.11e1,
                    "2018":1.91e1
                }
            }
        }
    ]
}

In Wolfram Language I can

a = GeneralUtilities`ToAssociations@ImportString[x, "JSON"]
<|
 "Dims1" ->
  {
   <|"Apple" ->
     <|"Baking" -> {"Pie", "Tart"}, "Plant" -> "Tree",
      "Tons" -> <|"2017" -> 12.3, "2018" -> 11.2|>|>
    |>,
   <|"Tomato" -> 
     <|"Cooking" -> {"Stew", "Sauce"}, "Plant" -> "Vine",
      "Tons" -> <|"2017" -> 8.1, "2018" -> 8.3|>|>
    |>,
   <|"Banana" ->
     <|"Name" -> "Banana", "Baking" -> {"Bread"}, 
      "Cooking" -> {"Fried"}, "Plant" -> "Arborescent",
      "Tons" -> <|"2017" -> 0.8, "2018" -> 0.5|>|>
    |>
   },
 "Dims2" ->
  {
   <|"Apple" ->
     <|"Name" -> "Apple", "Baking" -> {"Pie", "Tart"}, 
      "Plant" -> "Tree",
      "Tons" -> <|"2017" -> 13.1, "2018" -> 10.1|>|>
    |>,
   <|"Sweet Potato" ->
     <|"Cooking" -> {"Fried", "Steamed"}, "Baking" -> {"Pie"}, 
      "Plant" -> "Vine",
      "Tons" -> <|"2017" -> 11.1, "2018" -> 19.1|>|>
    |>
   }
 |>

and then with Query

Query[All, All, All, {"Baking"}]@a
<|"Dims1" -> 
   {<|"Apple" -> <|"Baking" -> {"Pie", "Tart"}|>|>, 
    <|"Tomato" -> <|"Baking" -> Missing["KeyAbsent", "Baking"]|>|>, 
    <|"Banana" -> <|"Baking" -> {"Bread"}|>|>}, 
  "Dims2" -> 
   {<|"Apple" -> <|"Baking" -> {"Pie", "Tart"}|>|>, 
    <|"Sweet Potato" -> <|"Baking" -> {"Pie"}|>|>}
|>

and include functions such as

Query[All, Join /* Flatten /* DeleteDuplicates, Values, "Baking" /* DeleteMissing]@a
<|"Dims1" -> {"Pie", "Tart", "Bread"}, "Dims2" -> {"Pie", "Tart"}|>

and

Query[All, Merge[Total] /* DateListPlot, All, "Tons", 
  KeyMap[DateObject[{FromDigits@#}, "Year"] &]]@a

enter image description here

How is this done with JSON in Python?

$\endgroup$
7
  • $\begingroup$ There is no such a thing in Python out of the box. $\endgroup$
    – Istvan
    Commented Jan 28, 2019 at 14:42
  • $\begingroup$ @Istvan Is there a package that enables such functionality? Perhaps loading the JSON into a hierarchical pandas.DataFrame; if such a thing exists. $\endgroup$
    – Edmund
    Commented Jan 29, 2019 at 10:28
  • $\begingroup$ @Istvan I found this post (15306448) on the pynq package. However, this seems limited to selection only and not able to apply or map function chains at specific levels while selecting. Is there something that can work with this package to add mapping/applying function chains along the levels while selecting? $\endgroup$
    – Edmund
    Commented Feb 4, 2019 at 16:28
  • $\begingroup$ I am not familiar anything in Python that resembles what you need. I need that Clojure Spectre does something similar but no Python. $\endgroup$
    – Istvan
    Commented Feb 8, 2019 at 10:05
  • $\begingroup$ @Istvan So how does one navigate, operation on, and visualise hierarchical datasets and subsets with Python? $\endgroup$
    – Edmund
    Commented Feb 8, 2019 at 11:52

1 Answer 1

0
$\begingroup$

ObjectPath is a query language for semi-structured data, including JSON, and has a Python API.

$\endgroup$
3
  • $\begingroup$ Thanks, for your answer. This looks somewhat promising except that it appears the the ObjectPath is in Global. I have multiple hierarchical JSON sources that need to be loaded at the same time. Also, it only appears to work with the specific functions that have been coded for it. I need something as flexible as Query which works with any built-in/package/user defined function available. ObjectPath does not seem able to allow for these two requirements $\endgroup$
    – Edmund
    Commented Feb 4, 2019 at 17:25
  • $\begingroup$ In Python, you can have multiple, independent json objects in memory at the same time. See document pypi.org/project/objectpath $\endgroup$ Commented Feb 4, 2019 at 19:18
  • $\begingroup$ Ah, the Tree function. I was going from the reference page where that function is not presented and it appears that everything is running from a Global $ operator. The arbitrary function chain applied at different levels of the hierarchy requirement is all that remains. Thanks. $\endgroup$
    – Edmund
    Commented Feb 4, 2019 at 20:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.