1
$\begingroup$

The Wolfram Language has a Query function that can traverse data structures and apply functions at different levels of the structure. I am working with multi-level JSON structures and need a function that has similar functionality as that of Query in the Wolfram Language.

Which Python package and function(s) best replicates this?

For a minimal working example, say I have the following JSON structure. (String escapes omitted for simplicity)

x = {
    "Dims1":[
        {
            "Apple":{
                "Baking":[
                    "Pie",
                    "Tart"
                ],
                "Plant":"Tree",
                "Tons":{
                    "2017":1.23e1,
                    "2018":1.12e1
                }
            }
        },
        {
            "Tomato":{
                "Cooking":[
                    "Stew",
                    "Sauce"
                ],
                "Plant":"Vine",
                "Tons":{
                    "2017":8.1,
                    "2018":8.3
                }
            }
        },
        {
            "Banana":{
                "Name":"Banana",
                "Baking":[
                    "Bread"
                ],
                "Cooking":[
                    "Fried"
                ],
                "Plant":"Arborescent",
                "Tons":{
                    "2017":0.8,
                    "2018":0.5
                }
            }
        }
    ],
    "Dims2":[
        {
            "Apple":{
                "Name":"Apple",
                "Baking":[
                    "Pie",
                    "Tart"
                ],
                "Plant":"Tree",
                "Tons":{
                    "2017":1.31e1,
                    "2018":1.01e1
                }
            }
        },
        {
            "Sweet Potato":{
                "Cooking":[
                    "Fried",
                    "Steamed"
                ],
                "Baking":[
                    "Pie"
                ],
                "Plant":"Vine",
                "Tons":{
                    "2017":1.11e1,
                    "2018":1.91e1
                }
            }
        }
    ]
}

In Wolfram Language I can

a = GeneralUtilities`ToAssociations@ImportString[x, "JSON"]
<|
 "Dims1" ->
  {
   <|"Apple" ->
     <|"Baking" -> {"Pie", "Tart"}, "Plant" -> "Tree",
      "Tons" -> <|"2017" -> 12.3, "2018" -> 11.2|>|>
    |>,
   <|"Tomato" -> 
     <|"Cooking" -> {"Stew", "Sauce"}, "Plant" -> "Vine",
      "Tons" -> <|"2017" -> 8.1, "2018" -> 8.3|>|>
    |>,
   <|"Banana" ->
     <|"Name" -> "Banana", "Baking" -> {"Bread"}, 
      "Cooking" -> {"Fried"}, "Plant" -> "Arborescent",
      "Tons" -> <|"2017" -> 0.8, "2018" -> 0.5|>|>
    |>
   },
 "Dims2" ->
  {
   <|"Apple" ->
     <|"Name" -> "Apple", "Baking" -> {"Pie", "Tart"}, 
      "Plant" -> "Tree",
      "Tons" -> <|"2017" -> 13.1, "2018" -> 10.1|>|>
    |>,
   <|"Sweet Potato" ->
     <|"Cooking" -> {"Fried", "Steamed"}, "Baking" -> {"Pie"}, 
      "Plant" -> "Vine",
      "Tons" -> <|"2017" -> 11.1, "2018" -> 19.1|>|>
    |>
   }
 |>

and then with Query

Query[All, All, All, {"Baking"}]@a
<|"Dims1" -> 
   {<|"Apple" -> <|"Baking" -> {"Pie", "Tart"}|>|>, 
    <|"Tomato" -> <|"Baking" -> Missing["KeyAbsent", "Baking"]|>|>, 
    <|"Banana" -> <|"Baking" -> {"Bread"}|>|>}, 
  "Dims2" -> 
   {<|"Apple" -> <|"Baking" -> {"Pie", "Tart"}|>|>, 
    <|"Sweet Potato" -> <|"Baking" -> {"Pie"}|>|>}
|>

and include functions such as

Query[All, Join /* Flatten /* DeleteDuplicates, Values, "Baking" /* DeleteMissing]@a
<|"Dims1" -> {"Pie", "Tart", "Bread"}, "Dims2" -> {"Pie", "Tart"}|>

and

Query[All, Merge[Total] /* DateListPlot, All, "Tons", 
  KeyMap[DateObject[{FromDigits@#}, "Year"] &]]@a

enter image description here

How is this done with JSON in Python?

$\endgroup$
  • $\begingroup$ There is no such a thing in Python out of the box. $\endgroup$ – Istvan Jan 28 at 14:42
  • $\begingroup$ @Istvan Is there a package that enables such functionality? Perhaps loading the JSON into a hierarchical pandas.DataFrame; if such a thing exists. $\endgroup$ – Edmund Jan 29 at 10:28
  • $\begingroup$ @Istvan I found this post (15306448) on the pynq package. However, this seems limited to selection only and not able to apply or map function chains at specific levels while selecting. Is there something that can work with this package to add mapping/applying function chains along the levels while selecting? $\endgroup$ – Edmund Feb 4 at 16:28
  • $\begingroup$ I am not familiar anything in Python that resembles what you need. I need that Clojure Spectre does something similar but no Python. $\endgroup$ – Istvan Feb 8 at 10:05
  • $\begingroup$ @Istvan So how does one navigate, operation on, and visualise hierarchical datasets and subsets with Python? $\endgroup$ – Edmund Feb 8 at 11:52
0
$\begingroup$

ObjectPath is a query language for semi-structured data, including JSON, and has a Python API.

$\endgroup$
  • $\begingroup$ Thanks, for your answer. This looks somewhat promising except that it appears the the ObjectPath is in Global. I have multiple hierarchical JSON sources that need to be loaded at the same time. Also, it only appears to work with the specific functions that have been coded for it. I need something as flexible as Query which works with any built-in/package/user defined function available. ObjectPath does not seem able to allow for these two requirements $\endgroup$ – Edmund Feb 4 at 17:25
  • $\begingroup$ In Python, you can have multiple, independent json objects in memory at the same time. See document pypi.org/project/objectpath $\endgroup$ – Brian Spiering Feb 4 at 19:18
  • $\begingroup$ Ah, the Tree function. I was going from the reference page where that function is not presented and it appears that everything is running from a Global $ operator. The arbitrary function chain applied at different levels of the hierarchy requirement is all that remains. Thanks. $\endgroup$ – Edmund Feb 4 at 20:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.