People who code: we want your input. Take the Survey

Hot answers tagged

53

Stolen from: https://stackoverflow.com/questions/33743978/spark-union-of-multiple-rdds Outside of chaining unions this is the only way to do it for DataFrames. from functools import reduce # For Python 3.x from pyspark.sql import DataFrame def unionAll(*dfs): return reduce(DataFrame.unionAll, dfs) unionAll(td2, td3, td4, td5, td6, td7, td8, td9, ...


18

You can manage Spark memory limits programmatically (by the API). As SparkContext is already available in your Notebook: sc._conf.get('spark.driver.memory') You can set as well, but you have to shutdown the existing SparkContext first: conf = SparkConf().setAppName("App") conf = (conf.setMaster('local[*]') .set('spark.executor.memory', '4G') ...


15

Sometime, when the dataframes to combine do not have the same order of columns, it is better to df2.select(df1.columns) in order to ensure both df have the same column order before the union. import functools def unionAll(dfs): return functools.reduce(lambda df1,df2: df1.union(df2.select(df1.columns)), dfs) Example: df1 = spark.createDataFrame([[1,1],...


15

"How can I import a .csv file into pyspark dataframes ?" -- there are many ways to do this; the simplest would be to start up pyspark with Databrick's spark-csv module. You can do this by starting pyspark with pyspark --packages com.databricks:spark-csv_2.10:1.4.0 then you can follow the following steps: from pyspark.sql import SQLContext sqlContext = ...


14

This can be done using StringIndexer in PySpark and the reverse using IndexToString for reference please check this: from pyspark.ml.feature import StringIndexer df = sqlContext.createDataFrame( [(0, "a"), (1, "b"), (2, "c"), (3, "a"), (4, "a"), (5, "c")], ["id", "category"]) indexer = StringIndexer(inputCol="category", outputCol="categoryIndex") ...


12

It will not expire until Spark is out of memory, at which point it will remove RDDs from cache which are used least often. When you ask for something that has been uncached it will recalculate the pipeline and put it in cache again. If this would be too expensive, unpersist other RDDs, don't cache them in the first place or persist them on your file system.


12

IPython has now moved to version 4.0, which means that if you are using it, it will be reading its configuration from ~/.jupyter, not ~/.ipython. You have to create a new configuration file with jupyter notebook --generate-config and then edit the resulting ~/.jupyter/jupyter_notebook_config.py file according to your needs. More installation instructions ...


11

The default value for spark.sql.shuffle.partitions is 200, and configures the number of partitions that are used when shuffling data for joins or aggregations. dataframe.repartition('id') creates 200 partitions with ID partitioned based on Hash Partitioner. Dataframe Row's with the same ID always goes to the same partition. If there is DataSkew on some ID's,...


10

Most often serialization error in (Py)Spark means that some part of your distributed code (e.g. functions passed to map) has dependencies on non-serializable data. Consider following example: rdd = sc.parallelize(range(5)) rdd = rdd.map(lambda x: x + 1) rdd.collect() Here you have distributed collection and lambda function to send to all workers. Lambda ...


10

Lots of questions here. First, for a truly new user with no data, there is no way to use a recommender model. If you have literally no information on the user, the only thing you can do is provide some default recommendations. Of course, once you have any data, and you can rebuild the model to incorporate the user, you can make recommendations. You can do ...


9

First of all let me tell you that I'm not a Spark expert; I've been using it quite a lot in the last few months, and I believe I now understand it, but I may be wrong. So, answering your questions: a.) they are equivalent, but not in the way you're seeing it; Spark will not optimize the graph if you are wondering, but the customMapper will still be ...


7

Probability can be found for the test dataset once you trained the model and transformed for the test dataset e.g: if your trained Naive Bayes model is model then model.transform(test) contains a node of probability, for more details please check the below code, going to show you the probability node and others useful nodes also for iris dataset. ...


7

For your recommendation engine, if you've chosen to go by item similarity approach, then you can use Spark's RowMatrix datatype to achieve this task. Item similarity approach is just about creating a square matrix of items in your catalog (i.e. itemID X itemID), where each element of the matrix is the magnitude of similarity between and . This ...


6

In addition to Jan's answer, I would like to point out that serialized RDD storage(/caching) works much better than normal RDD caching for large datasets. It also helps optimize garbage collection, in case of large datasets. Additionally, from the spark docs: When your objects are still too large to efficiently store despite this tuning, a much ...


6

I solved it by creating a spark-defaults.conf file in apache-spark/1.5.1/libexec/conf/ and adding the following line to it: spark.driver.memory 14g That solved my issue. But then I ran into another issue of exceeding max result size of 1024MB. The solution was to add another line in the file above: spark.driver.maxResultSize 2g


6

There is no need for pandas module to be installed because your data is generally stored in spark RDD or spark dataframes objects. The only interest I have found using Spark with pandas is when you want to load a local CSV / Excel dataset and then transform it into a spark dataframe. The "createDataFrame" method handles this approach. >>> ...


5

An RDD is a Read-only partition collection of records. RDD is or was the fundamental data structure of Spark. It allows a programmer to perform in-memory computations on large clusters in a fault-tolerant manner. Thus, speed up the task. A dataframe on the other hand organizes data into named columns. It is an immutable distributed collection of data. ...


5

First, when working with big data most of the time it's more convenient to work with a random subset rather than the whole thing: usually during the design and testing stages there is no need to work with the full data since optimal performance is not needed. Second, it's often useful to do an ablation study in order to check that using the full data is ...


4

Assume your configure file is ~/.ipython/profile_pyspark/ipython_notebook_config.py, you can still use this configure file by: ipython notebook --config='~/.ipython/profile_pyspark/ipython_notebook_config.py' or jupyter-notebook --config='~/.ipython/profile_pyspark/ipython_notebook_config.py'


4

You could try the following, testPassengerID = test.select('PassengerID').rdd this would select the column PassengerID and convert it into a rdd


4

You can use the SQL interface to get what you want: > df.selectExpr("from_utc_timestamp(start_time, tz) as testthis").show() +--------------------+ | testthis| +--------------------+ |2015-01-01 16:59:...| |2015-01-02 18:00:...| |2015-01-02 17:59:...| |2015-03-02 07:59:...| |2015-03-16 08:15:...| |2015-10-02 11:59:...| |2015-11-16 10:58:...| |...


4

How about using recursion? def union_all(dfs): if len(dfs) > 1: return dfs[0].unionAll(union_all(dfs[1:])) else: return dfs[0] td = union_all([td1, td2, td3, td4, td5, td6, td7, td8, td9, td10])


4

You are right, mllib uses RDDs and ml uses dataframes. At the beginning, there was only mllib because dataframes did not exist in spark. In fact, ml is kind of the new mllib, if you are new to spark, you should work with ml and dataframes.


4

I dont know which version you are using but I recommend DataFrames since most of upgrades are coming for DataFrames. (I prefer spark 2.3.2) First convert rdd to DataFrame: df = rdd.toDF(["M","Tu","W","Th","F","Sa","Su"]) Then select days you want to work with: df.select("M","W","F").show(3) Or directly use map with lambda: rdd.map(lambda x: [x[i] for i ...


3

You pass a function to the key parameter that it will virtually map your rows on to check for the maximum value. In this case you pass the str function which converts your floats to strings. Since '5.0' > '14.0' due to the nature of string comparisons, this is returned. What is usually a more likely use is using the key parameter as follows: test = sc....


3

Assuming you have an RDD each row of which is of the form (passenger_ID, passenger_name), you can do rdd.map(lambda x: x[0]). This is for a basic RDD If you use Spark sqlcontext there are functions to select by column name.


3

'RDD' object has no attribute 'select' This means that test is in fact an RDD and not a dataframe (which you are assuming it to be). Either you convert it to a dataframe and then apply select or do a map operation over the RDD. Please let me know if you need any help around this.


3

Just use the config option when setting SparkSession (as of 2.4) MAX_MEMORY = "5g" spark = SparkSession \ .builder \ .appName("Foo") \ .config("spark.executor.memory", MAX_MEMORY) \ .config("spark.driver.memory", MAX_MEMORY) \ .getOrCreate()


3

Assuming the rest of your configuration is correct all you have to do is to make spark-csv jar available to your program. There are a few ways you can achieve this: manually download required jars including spark-csv and csv parser (for example org.apache.commons.commons-csv) and put them somewhere on the CLASSPATH. using --packages option (use Scala ...


3

I'm fairly new to Spark, and have figured out how to integrate with with IPython on Windows 10 and 7. First, check your environment variables for Python and Spark. Here are mine: SPARK_HOME: C:\spark-1.6.0-bin-hadoop2.6\ I use Enthought Canopy, so Python is already integrated in my system path. Next, launch Python or IPython and use the following code. If ...


Only top voted, non community-wiki answers of a minimum length are eligible