Hot answers tagged

25

This is the import you need, and how to get the mean for a column named "RBIs": import org.apache.spark.sql.functions._ df.select(avg($"RBIs")).show() For the standard deviation, see scala - Calculate the standard deviation of grouped data in a Spark DataFrame - Stack Overflow For grouping by percentiles, I suggest defining a new column via a user-defined ...


11

Re: size of data The short answer Scala works for both small and large data, but its creation and development is motivated by needing something scalable. Scala is an acronym for “Scalable Language”. The long answer Scala is a functional programming language that runs on the jvm. The 'functional' part of this is a fundamental difference in the language ...


8

This is a bit off topic for this SE, or maybe opinion-based, but, I work in this field and I'd recommend Scala. No I would not characterize Scala as a "stats-oriented" Java. I'd describe it as what you get if you asked 3 people to design "Java 11" and then used all of their ideas at once. Java 8 remains great, but Scala fully embraces just about all the ...


6

In that link you posted, you can look at the python full solution here at the end and go through it to see what all is distributed. In short, some parts are distributed, like reading data from the file, but the very important parts like the distance computation are not. Running down, we see: sc = SparkContext("local[6]", "PythonKMeans") This ...


6

This is also returns average of column df.select(mean(df("ColumnName"))).show() +----------------+ | avg(ColumnName)| +----------------+ |230.522453845909| +----------------+


6

Hi (I created deeplearning4j: warning biased source), It's fairly new but we are open to feedback. We ported keras to scala: http://github.com/deeplearning4j/ScalNet/ This uses deeplearning4j underneath. We would love contributions or issues.


5

I think currently there is no pure scala deep learning library which can be compared with mxnet, caffe, keras and etc. For lang on JVM, u can try this one: https://deeplearning4j.org/


5

ScalaNLP is a suite of machine learning and numerical computing libraries with support for common natural language processing tasks. Here is a newly updated list of scala libraries for data science.


4

From listening to presentations by Martin Odersky, the creator of Scala, it is especially well suited for building highly scalable systems by leveraging functional programming constructs in conjuction with object orientation and flelxible syntax. It is also useful for development of small systems and rapid prototyping because it takes less lines of code than ...


4

First, the spark programming guide for LogisticRegressionWithSGD recommends using L-BFGS instead, so perhaps focus on the one. As for variable selection, the model description on the MLLib page for regressions has a nice explanation of how models are constructed and selected, but it does not address variable selection. This leads me to believe that it ...


4

Let's look at the error message: found : Array[org.apache.spark.mllib.linalg.Vector] required: org.apache.spark.mllib.linalg.Vector "found" is the type of object that it came across, "required" is the type of the object that the function accepts. The types look mostly the same ( org.apache.spark.mllib.linalg.Vector ) but the first is Array[X] and the ...


3

I changed my code to this: import org.apache.spark.SparkContext import org.apache.spark.SparkConf import org.apache.spark.sql.cassandra.CassandraSQLContext import org.apache.spark.rdd.RDD object Test2 { def calculate(numberAsString: String, testRDD: RDD[Int]): RDD[String] = { val newRDD = testRDD.map { x => numberAsString } newRDD } } object ...


3

Looking at another question on Stack Overflow about serialization exceptions in Spark, it says that anonymous functions serialize their containing class, and if that class contains the SparkContext -- which is not serializable -- then an error is thrown. Perhaps this is happening to you?


3

The other side of the coin: I don't have an extensive experience with Scala; I have written approximately 10,000 lines of Scala code. However, consider that Scala code is often much shorter than its rough equivalent of 40,000 lines of Java. On short I don't like Scala at all. I love it's goals, it's ideas but for production use I consider the ...


2

Checkout Breeze and apache commons math for the maths, and ScalaLab for some nice examples of how to plot things in Scala. I've managed to get an environment setup where this would just be a couple of lines. I dont actually use ScalaLab, rather borrow some of its code, I use Intellij worksheets instead.


2

I don't know about that specific implementation, but we use the mllib k-means here at my work, to some degree of success. It is distributed and runs on Spark!


2

Vegas is a library that strives to substitute for matplotlib in Zeppelin/Spark environments. It works on Spark DataFrames.


2

You're close! You're just missing a .rdd! Try this: df.groupBy("Product_ID").agg(collect_list($"Stock")).rdd.saveAsTextFile("PATH/results.csv")


2

This is not standard part of the API of DataFrames. You can either map it to a RDD, join the row entries to a string and save that or the more flexible way is to use the DataBricks spark-csv package that can be found here. If it's just one column you can map it to a RDD and just call .saveAsTextFile(filename)


2

The two languages have pretty similar benefits since Scala can call Java libraries. So Java machine learning packages like Weka (http://www.cs.waikato.ac.nz/ml/weka/), can in theory be easily used with Scala. There are minor pros and cons to each, however: Java is a language that most software engineers with 5+ years of experience understand. If you go to ...


2

You can do the following: A Pipeline can be made of other pipelines. Isn't that great? A Pipeline inherit from the Estimator class and by definition, a PipelineStage can be either an Estimator or a Transformer. This way, you can build smaller pipelines, save them separately and on the other software/class, join them again as a single one and call transform ...


2

I have done only a little bit of scala (spark streaming for database replication), but have worked with python for about a year. I would recommend python for the following reasons: We do a lot of notebook centric development at my job. I would recommend that because the Jupyter IDE allows you to visualize/debug your data really effectively, which is ...


2

I am using both almond and toree. For executing only scala code, I am using almond. If you want to learn new features in scala (ex: 2.13), then use almond(Add Scala 2.13.4 support in almond v0.11.0). Almond might start supporting scala 3.0 once it is released. For spark, I am using toree. Toree works fine for me. I am using toree 0.5.0 with spark 3.0.0. ...


1

For the first you can do as follow : val discount = salesdata.map( str => str.split(",")) .map( array => (array(0), array(1), array(2), array(3).toDouble) ) .map{ case(a, b, c, d) => (a, b, c, d-0.3*d)} I'm not sure to understand the second, this will gives you the min and max per c-itemID val ...


1

After reading the source code, I find out that the problem is nothing related to your memory, but the limit of integer. In the source code, it shows both of $n\min(2k,n)$ and $\min(2k,n)*(\min(2k,n)+8)$ should be less than Integer.MAX_VALUE, which is $2^{31}-1=2147483647$. In your case, $n\min(2k,n)=36176793968>2147483647$, and $\min(2k,n)*(\min(2k,n)+8)=...


1

Use this syntax: val joinedDF = students_positive_grande.as('a).join( df.as('b), $"a.Customer_ID" === $"b.Customer_ID") joinedDF.select($"a.Customer_ID", $"b.Customer_ID")


1

Let's start with your problem definition: "a good strategy make the relationships first and then count the occurrences". That is, roughly, the basic strategy that market basket analysis algorithms use. However, algorithms like Apriori or FPGrowth are specially designed to analyze such datasets (at scale) and infer the inherent association rules between ...


1

Usually, those non-sensical words are not problematic because they appear in one or two documents, and people usually just filter words with such low frequencies. Are you not able to do this? Or do your non-sensical words appear that much? Anyhow, your suggestion is what people use: outright ignore tokens that aren't discernable by a dictionary. Is ...


1

Broadcasting your data and learn on it with different learning parameters per spark partition is a solution only if your data isn t so big it can fit in each machine memory. If you desire apply ML models at scale you have to deal with subquadratic complexity, if not you will to increase number of nodes accordingly to the complexity, but its something ...


1

The way you can use scikit-learn is basically broadcasting your data to the workers and then do different folds of crossvalidation or different parameter settings in your grid_search on different workers. That is all that the scikit-learn package in pySpark does as far as I know. This is similar to a normal mapping. Implementing this should be relatively ...


Only top voted, non community-wiki answers of a minimum length are eligible