Questions tagged [scala]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
-1
votes
1answer
61 views

Which language to learn for Machine Learning? [closed]

I am currently working in BigData with Spark-Scala framework. I want to learn Machine learning from scratch. Which language would be better to learn for machine learning, Scala or Python?
1
vote
0answers
19 views

Is there a way to use a pom.xml file to update spark configuration?

I am trying to update my spark configuration to solve some dependency problems. This pom.xml file seems to be useful for this purpose. I am using a spark docker image. ...
0
votes
0answers
36 views

Building recommendation engine from transactions data

I am trying to build a recommendation engine for an e-commerce company and I have the following input files : 1) past user transactions + in-app events 2) a new list of campaigns I should recommend ...
0
votes
0answers
49 views

XGBoost not learning

I have developed a train set for XGBoost to apply a learning to rank function on top of with the following parameters: ...
0
votes
0answers
174 views

Spark Scala concatenate 2 different data frames

I have 2 different Spark data frames and I want to concatenate them together by columns with no join operations. How can I do it using Scala?
0
votes
0answers
44 views

scala spark FP-Growth no results displayed

I have implemented the FP-growth algorithm and it works fine with this sample data: r z h k p z y x w v u t s s x o n r x z y m t s q e z x z y r q t p when I ...
0
votes
2answers
46 views

Apache Spark alternatives for local compute

I am creating a relatively large hobby project in Scala that needs a few ml algorithms for text classification into topics. My dataset is not huge, it is < 500,000 items with dimensionality 5 (2 ...
1
vote
1answer
181 views

Standard Scaler drops accuracy significantly in Scala Spark

I am working on Scala with Spark for a prediction model. I tried both Normalization and Standard Scaling and both of them drops my accuracy significantly. Without the accuracy is ~90% (on training ...
2
votes
1answer
103 views

XGBoost most important features appear in multiple trees multiple times

I am fitting xgboost model (scala-spark) to my dataset of transactions. I have about 2 millions of transactions in my training set which is highly unbalanced with a ratio of positive/negative<0.001 ...
0
votes
1answer
38 views

How does “sameModel” mean in Spark ML Pipeline, (Section: Example: Pipeline), from the docs?

I'm looking at the Spark ML docs in scala for Section: Example Pipeline https://spark.apache.org/docs/latest/ml-pipeline.html#example-pipeline. From the example, the model is fit using a pipeline (...
1
vote
1answer
14 views

Scala mllib 2.2.0 k means. Number of runs

As per the documentation here, the runs parameter does nothing. I have looked at the actual Scala code for k means and it appears that the ...
1
vote
1answer
2k views

How to apply StandardScaler and OneHotEncoder simultaneously in Spark Machine learning?

I try to create a machine learning model, linear regression, to predict a price of a diamonds. All examples that I found online do not have a step with scaling of data, using MinMaxScaler or ...
2
votes
1answer
284 views

Best practice for developing using Spark [closed]

I am looking for any tips and best practice on how to develop applications using Spark. I currently have access to a cluster, with data as well as a version of Spark 2.1.0 on an edge node and IntelliJ ...
0
votes
1answer
170 views

How to Access a Text File from IBM DSX Scala-Spark Notebook [closed]

I have recently faced problem in accessing text files that I have uploaded from my local computer to the IBM DSX Cloud Object Storage so that I can use it in my Spark-Scala notebooks. It appeared to ...
1
vote
1answer
60 views

Multidimensional regression in Scala

I have a queuing model written in Scala where different categories of people end up a different queues. We have a dataset providing a map of features to the numbers of people ending up at each queue, ...
5
votes
1answer
887 views

Saving Large Spark ML Pipeline to HDFS

I'm having trouble saving a large (relative to spark.rpc.message.maxSize) Spark ML pipeline to HDFS. Specifically, when I try to save the model to HDFS, it gives me an error related to spark's maximum ...
2
votes
0answers
55 views

Why does my master node get heap memory full for inbuilt SVD API in Apache Spark during calculation of inverse of a square matrix?

I am using a three node system: Master node: $64$ GB RAM 2 slave node: $32$ GB RAM I am calculating the inverse of a square matrix using inbuilt Apache Spark function SVD. When I called SVD function ...
1
vote
0answers
47 views

How to adapt the LBFGS algorithm to accept a different data input parameter in Spark?

I´m trying to use the LBFGS algorithm to minimize the parameters of a model used for Customer Life Time Value analysis (https://github.com/CamDavidsonPilon/lifetimes). The lifetimes lib in python uses ...
-1
votes
1answer
2k views

Spark: calculating percentage from a non-Int field

I have a spreadsheet of banking information and one of the goals is to find out the failure rate of an advertising campaign. I think I need to get a count of the total number of entries vs. the total ...
0
votes
1answer
1k views

Scala RDD operation

I am new in scala. I have a csv file stored in hdfs. I am reading that file in scala using ...
1
vote
1answer
377 views

Issue with Spark SVD

I have the following dataset with the dimensions: Rows: 41174 Columns: 439316 The matrix is very sparse and on this, I want to perform Dimensionality Reduction. I am using Spark's computeSVD ...
4
votes
2answers
4k views

What is the best deep learning library for scala? [closed]

Does any one has a recommendation for what libraries to use for deep learning?
3
votes
2answers
464 views

Hashing trick with random forest in scala

I am trying to perform a hashing trick and then a random forest with scala. I have the following code: ...
0
votes
1answer
6k views

Join two dataframes - Spark Mllib

I've two dataframes. The first have the some details from all the students, and the second have only the students that haved positive grade. How can I return only the details of the student that have ...
2
votes
3answers
1k views

Task not serializable Error

...
2
votes
3answers
7k views

value saveAsTextFile is not a member of org.apache.spark.sql.DataFrame

How Can I save a DataFrame as a CSV in HDFS? I'm trying with this: df.groupBy("Product_ID").agg(collect_list($"Stock")).saveAsTextFile("PATH/results.csv")
0
votes
1answer
2k views

Apply GroupByKey to identify the Frequently Products Purchase Together

I've a data set with this fields: Transaction_ID Customer_ID Department Product_ID And I'm trying to obtain a tuple with the products associated with each ...
1
vote
2answers
1k views

Market Basket Analysis - Data Modelling

Imagine that I've the following dataset: ...
0
votes
1answer
1k views

Items in a transaction must be unique but got WrappedArray

I'm trying to do a algorithm in Spark Mllib. I'm trying to do a Market Basket Analysis. And I've as a data set this: ...
0
votes
1answer
970 views

How to down-weight non-words in text classification?

Background: Documents coming in as well as training set have gone through Apache Tika with Tesseract for inline images. This works well, except when it doesn't. Many docs are old, scanned images ...
3
votes
3answers
3k views

Scala vs Java if you're NOT going to use Spark?

I'm facing some indecision when choosing how to allocate my scarce learning time for the next few months between Scala and Java. I would like help objectively understanding the practical tradeoffs. ...
2
votes
2answers
426 views

Spark Scala alternative Machine Learning Library?

I’m using Spark Scala for multiclass classification, and features are continuous. MLlib seems to be limited to Decision Tree and Random Forest for this type of classification – for Naïve Bayes, ...
5
votes
2answers
3k views

Distributed k-means in Spark

I want to implement K-means algorithm in Spark. I am looking for a starting point and I found Berkeley's naive implementation. However, is that distributed? I mean I see no mapreduce operations. Or ...
3
votes
1answer
637 views

Sampling with replacement, specify the probabilities

I am trying to do sampling with replacement in Scala/Spark, defining the probabilities for each class. This is how I would do it in R. ...
4
votes
1answer
2k views

Plotting libraries for Scala on Zeppelin

My main question is it looks like Zeppelin limit the display of the results to on 1000, I know that I can change this number but when I change it Zeppelin become slow. And it looks like the default ...
13
votes
2answers
62k views

How to calculate the mean of a dataframe column and find the top 10%

I am very new to Scala and Spark, and am working on some self-made exercises using baseball statistics. I am using a case class create a RDD and assign a schema to the data, and am then turning it ...
4
votes
2answers
4k views

How Mllib in Spark select variables in logistic regression

I have a question about MLlib in Spark.(with Scala) I'm trying to understand how LogisticRegressionWithLBFGS and LogisticRegressionWithSGD work. I usually use SAS or R to do logistic regressions but ...
15
votes
4answers
502 views

Data Science Tools Using Scala

I know that Spark is fully integrated with Scala. It's use case is specifically for large data sets. Which other tools have good Scala support? Is Scala best suited for larger data sets? Or is it also ...
4
votes
1answer
807 views

How to set up multi cluster spark without hadoop on Google Compute engine

I'm new to apache spark. Is it possible to configure multi cluster spark without hadoop? If so, can you please provide the steps. I would like to create clusters on Google Compute Engine (1-...
6
votes
3answers
1k views

Summarize and visualize a CSV in Java/Scala?

I would like to summarize (as in R) the contents of a CSV (possibly after loading it, or storing it somewhere, that's not a problem). The summary should contain the quartiles, mean, median, min and ...