What's the deal with Deno? We talk with a major contributor to find out. Listen now.
45

Stolen from: https://stackoverflow.com/questions/33743978/spark-union-of-multiple-rdds Outside of chaining unions this is the only way to do it for DataFrames. from functools import reduce # For Python 3.x from pyspark.sql import DataFrame def unionAll(*dfs): return reduce(DataFrame.unionAll, dfs) unionAll(td2, td3, td4, td5, td6, td7, td8, td9, ...


43

Hadoop means HDFS, YARN, MapReduce, and a lot of other things. Do you mean Spark vs MapReduce? Because Spark runs on/with Hadoop, which is rather the point. The primary reason to use Spark is for speed, and this comes from the fact that its execution can keep data in memory between stages rather than always persist back to HDFS after a Map or Reduce. This ...


22

This is the import you need, and how to get the mean for a column named "RBIs": import org.apache.spark.sql.functions._ df.select(avg($"RBIs")).show() For the standard deviation, see scala - Calculate the standard deviation of grouped data in a Spark DataFrame - Stack Overflow For grouping by percentiles, I suggest defining a new column via a user-defined ...


17

You can manage Spark memory limits programmatically (by the API). As SparkContext is already available in your Notebook: sc._conf.get('spark.driver.memory') You can set as well, but you have to shutdown the existing SparkContext first: conf = SparkConf().setAppName("App") conf = (conf.setMaster('local[*]') .set('spark.executor.memory', '4G') ...


14

This can be done using StringIndexer in PySpark and the reverse using IndexToString for reference please check this: from pyspark.ml.feature import StringIndexer df = sqlContext.createDataFrame( [(0, "a"), (1, "b"), (2, "c"), (3, "a"), (4, "a"), (5, "c")], ["id", "category"]) indexer = StringIndexer(inputCol="category", outputCol="categoryIndex") ...


12

IPython has now moved to version 4.0, which means that if you are using it, it will be reading its configuration from ~/.jupyter, not ~/.ipython. You have to create a new configuration file with jupyter notebook --generate-config and then edit the resulting ~/.jupyter/jupyter_notebook_config.py file according to your needs. More installation instructions ...


12

I don't think you necessarily need to convert the individual log entries into vectors for use in an algorithm. I would guess that what you are interested in is a sequence of log entries, which represent a series of events, ordered in time, which together make up a series of 'cases'. Here the relationship between a series of collected log entries is ...


12

It will not expire until Spark is out of memory, at which point it will remove RDDs from cache which are used least often. When you ask for something that has been uncached it will recalculate the pipeline and put it in cache again. If this would be too expensive, unpersist other RDDs, don't cache them in the first place or persist them on your file system.


10

Most often serialization error in (Py)Spark means that some part of your distributed code (e.g. functions passed to map) has dependencies on non-serializable data. Consider following example: rdd = sc.parallelize(range(5)) rdd = rdd.map(lambda x: x + 1) rdd.collect() Here you have distributed collection and lambda function to send to all workers. Lambda ...


10

Lots of questions here. First, for a truly new user with no data, there is no way to use a recommender model. If you have literally no information on the user, the only thing you can do is provide some default recommendations. Of course, once you have any data, and you can rebuild the model to incorporate the user, you can make recommendations. You can do ...


9

First of all let me tell you that I'm not a Spark expert; I've been using it quite a lot in the last few months, and I believe I now understand it, but I may be wrong. So, answering your questions: a.) they are equivalent, but not in the way you're seeing it; Spark will not optimize the graph if you are wondering, but the customMapper will still be ...


9

Sometime, when the dataframes to combine do not have the same order of columns, it is better to df2.select(df1.columns) in order to ensure both df have the same column order before the union. import functools def unionAll(dfs): return functools.reduce(lambda df1,df2: df1.union(df2.select(df1.columns)), dfs) Example: df1 = spark.createDataFrame([[1,...


8

There's a related example to your problem in the Spark repo here. The strategy is to represent the documents as a RowMatrix and then use its columnSimilarities() method. That will get you a matrix of all the cosine similarities. Extract the row which corresponds to your query document and sort. That will give the indices of the most-similar documents. ...


8

As per your problem, I think it might be easier to use lit. Try this- from pyspark.sql.functions import lit new_df = df.withColumn('column_name', lit(1)) Hope it helps!


8

Using lit would convert all values of the column to the given value. To do it only for non-null values of dataframe, you would have to filter non-null values of each column and replace your value. when can help you achieve this. from pyspark.sql.functions import when df.withColumn('c1', when(df.c1.isNotNull(), 1)) .withColumn('c2', when(df.c2....


8

The default value for spark.sql.shuffle.partitions is 200, and configures the number of partitions that are used when shuffling data for joins or aggregations. dataframe.repartition('id') creates 200 partitions with ID partitioned based on Hash Partitioner. Dataframe Row's with the same ID always goes to the same partition. If there is DataSkew on some ID's,...


7

The Dataframe Python API exposes the RDD of a Dataframe by calling the following : df.rdd # you can save it, perform transformations of course, etc. df.rdd returns the content as an pyspark.RDD of Row. You can then map on that RDD of Row transforming every Row into a numpy vector. I can't be more specific about the transformation since I don't know what ...


7

Spark is intended to be pointed at large distributed data sets, so as you suggest, the most typical use cases will involve connecting to some sort of Cloud system like AWS. In fact, if the data set you aim to analyze can fit on your local system, you'll usually find that you can analyze it just as simply using pure python. If you're trying to leverage a ...


7

Probability can be found for the test dataset once you trained the model and transformed for the test dataset e.g: if your trained Naive Bayes model is model then model.transform(test) contains a node of probability, for more details please check the below code, going to show you the probability node and others useful nodes also for iris dataset. ...


7

For your recommendation engine, if you've chosen to go by item similarity approach, then you can use Spark's RowMatrix datatype to achieve this task. Item similarity approach is just about creating a square matrix of items in your catalog (i.e. itemID X itemID), where each element of the matrix is the magnitude of similarity between and . This ...


6

If I were you, I would pick anyone of the frameworks I am comfortable with and implement the use-case. Spark-Streaming + MLlib should work and would be my choice since its user base is on the rise and it is one of the most popular project under the Apache Umbrella with good enterprise business plan. Both Cloudera and Hortonworks provide enterprise level ...


6

A quick glance at the docs for LogisticRegressionWithLBFGS indicates that it uses feature scaling and L2-Regularization by default. I suspect that R's glm is returning a maximum likelihood estimate of the model while Spark's LogisticRegressionWithLBFGS is returning a regularized model estimate. Note how the estimated model weights of the Spark model are all ...


6

I solved it by creating a spark-defaults.conf file in apache-spark/1.5.1/libexec/conf/ and adding the following line to it: spark.driver.memory 14g That solved my issue. But then I ran into another issue of exceeding max result size of 1024MB. The solution was to add another line in the file above: spark.driver.maxResultSize 2g


6

In that link you posted, you can look at the python full solution here at the end and go through it to see what all is distributed. In short, some parts are distributed, like reading data from the file, but the very important parts like the distance computation are not. Running down, we see: sc = SparkContext("local[6]", "PythonKMeans") This ...


6

In addition to Jan's answer, I would like to point out that serialized RDD storage(/caching) works much better than normal RDD caching for large datasets. It also helps optimize garbage collection, in case of large datasets. Additionally, from the spark docs: When your objects are still too large to efficiently store despite this tuning, a much ...


6

The problem you are facing is a time series problem. Your events are categorial which is a specific case (so most common techniques like arima and Fourier transform are irrelevant). Before getting into the analysis, try to find out whether the events among nodes are independent. If they are independent, you can break them into sequences per node and analyze ...


6

You have partly answered this question yourself ("because converting to integers implies that there is an ordering between features"). I will just clarify the terminology a bit more. Categorical data: information has categories, but no natural ordering defined between them (gender, name of user's cat) Ordinal data: information has categories with natural ...


5

This is also returns average of column df.select(mean(df("ColumnName"))).show() +----------------+ | avg(ColumnName)| +----------------+ |230.522453845909| +----------------+


5

There is a recently-added feature in Spark 2.2.0... spark.read.csv(file, multiLine=True) https://issues.apache.org/jira/browse/SPARK-19610 https://issues.apache.org/jira/browse/SPARK-20980


5

The question is more related to Apache Spark architecture and map reduce; there are more than one questions here, however, the central piece of your question perhaps is For example, one of the means to determine PCs of a data is to calculate covariance matrix of the features. When using HDFS based architecture for example, the original data is distributed ...


Only top voted, non community-wiki answers of a minimum length are eligible