Questions tagged [scala]
Scala is a high level language that combines functional and object oriented programming with high performance runtimes. Scala programming language is build to implement scale able solutions to crunch big data / data science in order to produce actionable insights is a great languages for large-scale projects. Designed to express common programming patterns in a concise, elegant, and type-safe way, it fuses both imperative and functional programming styles.
48
questions
0
votes
0
answers
96
views
reading data from Azure Event Hub with Databricks
I'm having problems reading the data from event hub.
questions :
what am i doing wrong in connecting to event hub using databricks?
is the problem with the message? or is it in the script?
is the ...
- 1
0
votes
0
answers
18
views
How to reference an intermediary object in scala?
I am using spark/scala to do some transformations on a spark dataframe. I have the following code:
...
- 11
0
votes
1
answer
177
views
IllegalArgumentException at org.apache.xbean.asm5.ClassReader.<init>(Unknown Source) when training an ALS implementation of spark in scala
I was following this tutorial trying to write a collaborative recommender system using the alternating least squares algorithm in spark. I am using the movie lens dataset which can be found here.
My ...
- 11
1
vote
1
answer
273
views
Outlier Elimination in Spark With InterQuartileRange Results in Error
I have the following function that is supposed to calculate the outlier for a given dataset.
...
- 137
1
vote
2
answers
102
views
unsupervised anomaly detection for univariate fast frequency time series data?
I have a univariate time series (there is a value for each time sampling) (sampling time: 66.66 micro second, number of samples/sampling time=151) coming from a scala customer
This time series ...
- 1,794
0
votes
1
answer
27
views
determining size of batch, time of sending and memory in to send from scala to ML section
I have a time series (sampling time: 66.66 micro second, number of samples/sampling time=151), I would like to determine some anomalies in them, the inputs are made by scala customer message bus.
...
- 1,794
1
vote
0
answers
194
views
how to calculate the cosine similarity between two files?
I am using spark and scala to implement an issue. files contain phrases or sentences. I want to use domain based method to calculate the cosine similarity between tags.I convert two files into a ...
- 11
4
votes
1
answer
1k
views
Has anyone succeeded in finding a good Scala/Spark kernel for Jupyter?
The ones I've tried so far
Almond: Works very well for just Scala, but you have to import dependencies, and it gets tedious after a while. And unfortunately can't run when using Spark with YARN ...
- 141
1
vote
1
answer
552
views
Can I use model which is trained using Keras to Scala?
I train the NLP classification model using Keras by Python.
But to deploy the trained model to a platform I need to use Scala.
Can I use Scala with Keras model what I trained using Python?
Thanks
- 11
3
votes
3
answers
1k
views
How to install Polynote on Windows?
I've been searching around the Internet for a while but I have not been able to find detailed instructions on how to install Polynote (the polyglot notebook
with first-class Scala support) for Windows ...
- 3,550
-1
votes
1
answer
86
views
Which language to learn for Machine Learning? [closed]
I am currently working in BigData with Spark-Scala framework. I want to learn Machine learning from scratch.
Which language would be better to learn for machine learning, Scala or Python?
- 1
1
vote
0
answers
36
views
Is there a way to use a pom.xml file to update spark configuration?
I am trying to update my spark configuration to solve some dependency problems.
This pom.xml file seems to be useful for this purpose.
I am using a spark docker image.
...
- 425
1
vote
0
answers
290
views
XGBoost not learning
I have developed a train set for XGBoost to apply a learning to rank function on top of with the following parameters:
...
- 245
1
vote
0
answers
489
views
Spark Scala concatenate 2 different data frames
I have 2 different Spark data frames and I want to concatenate them together by columns with no join operations.
How can I do it using Scala?
- 531
0
votes
2
answers
603
views
Apache Spark alternatives for local compute
I am creating a relatively large hobby project in Scala that needs a few ml algorithms for text classification into topics. My dataset is not huge, it is < 500,000 items with dimensionality 5 (2 ...
- 103
1
vote
1
answer
382
views
Standard Scaler drops accuracy significantly in Scala Spark
I am working on Scala with Spark for a prediction model. I tried both Normalization and Standard Scaling and both of them drops my accuracy significantly.
Without the accuracy is ~90% (on training ...
- 3,880
2
votes
1
answer
259
views
XGBoost most important features appear in multiple trees multiple times
I am fitting xgboost model (scala-spark) to my dataset of transactions. I have about 2 millions of transactions in my training set which is highly unbalanced with a ratio of positive/negative<0.001 ...
- 121
0
votes
1
answer
49
views
How does "sameModel" mean in Spark ML Pipeline, (Section: Example: Pipeline), from the docs?
I'm looking at the Spark ML docs in scala for Section: Example Pipeline https://spark.apache.org/docs/latest/ml-pipeline.html#example-pipeline. From the example, the model is fit using a pipeline (...
- 11
1
vote
1
answer
22
views
Scala mllib 2.2.0 k means. Number of runs
As per the documentation here, the runs parameter does nothing.
I have looked at the actual Scala code for k means and it appears that the ...
- 111
1
vote
1
answer
4k
views
How to apply StandardScaler and OneHotEncoder simultaneously in Spark Machine learning?
I try to create a machine learning model, linear regression, to predict a price of a diamonds.
All examples that I found online do not have a step with scaling of data, using MinMaxScaler or ...
2
votes
1
answer
385
views
Best practice for developing using Spark [closed]
I am looking for any tips and best practice on how to develop applications using Spark. I currently have access to a cluster, with data as well as a version of Spark 2.1.0 on an edge node and IntelliJ ...
- 488
0
votes
1
answer
206
views
How to Access a Text File from IBM DSX Scala-Spark Notebook [closed]
I have recently faced problem in accessing text files that I have uploaded from my local computer to the IBM DSX Cloud Object Storage so that I can use it in my Spark-Scala notebooks. It appeared to ...
- 19
1
vote
1
answer
121
views
Multidimensional regression in Scala
I have a queuing model written in Scala where different categories of people end up a different queues. We have a dataset providing a map of features to the numbers of people ending up at each queue, ...
- 111
5
votes
1
answer
2k
views
Saving Large Spark ML Pipeline to HDFS
I'm having trouble saving a large (relative to spark.rpc.message.maxSize) Spark ML pipeline to HDFS. Specifically, when I try to save the model to HDFS, it gives me an error related to spark's maximum ...
- 1,515
2
votes
0
answers
88
views
Why does my master node get heap memory full for inbuilt SVD API in Apache Spark during calculation of inverse of a square matrix?
I am using a three node system:
Master node: $64$ GB RAM
2 slave node: $32$ GB RAM
I am calculating the inverse of a square matrix using inbuilt Apache Spark function SVD. When I called SVD function ...
- 301
1
vote
0
answers
106
views
How to adapt the LBFGS algorithm to accept a different data input parameter in Spark?
I´m trying to use the LBFGS algorithm to minimize the parameters of a model used for Customer Life Time Value analysis (https://github.com/CamDavidsonPilon/lifetimes). The lifetimes lib in python uses ...
- 111
-1
votes
1
answer
3k
views
Spark: calculating percentage from a non-Int field
I have a spreadsheet of banking information and one of the goals is to find out the failure rate of an advertising campaign. I think I need to get a count of the total number of entries vs. the total ...
- 101
0
votes
2
answers
2k
views
Scala RDD operation
I am new in scala.
I have a csv file stored in hdfs. I am reading that file in scala using
...
- 165
1
vote
1
answer
542
views
Issue with Spark SVD
I have the following dataset with the dimensions:
Rows: 41174
Columns: 439316
The matrix is very sparse and on this, I want to perform Dimensionality Reduction. I am using Spark's computeSVD ...
- 133
5
votes
2
answers
5k
views
What is the best deep learning library for scala? [closed]
Does any one has a recommendation for what libraries to use for deep learning?
- 724
3
votes
2
answers
548
views
Hashing trick with random forest in scala
I am trying to perform a hashing trick and then a random forest with scala. I have the following code:
...
- 101
0
votes
1
answer
7k
views
Join two dataframes - Spark Mllib
I've two dataframes. The first have the some details from all the students, and the second have only the students that haved positive grade.
How can I return only the details of the student that have ...
- 173
3
votes
3
answers
2k
views
3
votes
3
answers
11k
views
value saveAsTextFile is not a member of org.apache.spark.sql.DataFrame
How Can I save a DataFrame as a CSV in HDFS?
I'm trying with this:
df.groupBy("Product_ID").agg(collect_list($"Stock")).saveAsTextFile("PATH/results.csv")
- 179
0
votes
1
answer
3k
views
Apply GroupByKey to identify the Frequently Products Purchase Together
I've a data set with this fields:
Transaction_ID
Customer_ID
Department
Product_ID
And I'm trying to obtain a tuple with the products associated with each ...
- 173
1
vote
2
answers
2k
views
0
votes
1
answer
2k
views
Items in a transaction must be unique but got WrappedArray
I'm trying to do a algorithm in Spark Mllib. I'm trying to do a Market Basket Analysis. And I've as a data set this:
...
- 173
0
votes
1
answer
1k
views
How to down-weight non-words in text classification?
Background:
Documents coming in as well as training set have gone through Apache Tika with Tesseract for inline images. This works well, except when it doesn't. Many docs are old, scanned images ...
- 135
4
votes
3
answers
4k
views
Scala vs Java if you're NOT going to use Spark?
I'm facing some indecision when choosing how to allocate my scarce learning time for the next few months between Scala and Java.
I would like help objectively understanding the practical tradeoffs.
...
- 1,919
2
votes
2
answers
474
views
Spark Scala alternative Machine Learning Library?
I’m using Spark Scala for multiclass classification, and features are continuous. MLlib seems to be limited to Decision Tree and Random Forest for this type of classification – for Naïve Bayes, ...
- 133
5
votes
2
answers
3k
views
Distributed k-means in Spark
I want to implement K-means algorithm in Spark. I am looking for a starting point and I found Berkeley's naive implementation. However, is that distributed?
I mean I see no mapreduce operations. Or ...
- 291
3
votes
1
answer
2k
views
Sampling with replacement, specify the probabilities
I am trying to do sampling with replacement in Scala/Spark, defining the probabilities for each class.
This is how I would do it in R.
...
- 31
4
votes
1
answer
2k
views
Plotting libraries for Scala on Zeppelin
My main question is it looks like Zeppelin limit the display of the results to on 1000, I know that I can change this number but when I change it Zeppelin become slow. And it looks like the default ...
- 594
14
votes
2
answers
105k
views
How to calculate the mean of a dataframe column and find the top 10%
I am very new to Scala and Spark, and am working on some self-made exercises using baseball statistics. I am using a case class create a RDD and assign a schema to the data, and am then turning it ...
- 243
4
votes
2
answers
5k
views
How Mllib in Spark select variables in logistic regression
I have a question about MLlib in Spark.(with Scala)
I'm trying to understand how LogisticRegressionWithLBFGS and LogisticRegressionWithSGD work. I usually use SAS or R to do logistic regressions but ...
- 113
15
votes
4
answers
601
views
Data Science Tools Using Scala
I know that Spark is fully integrated with Scala. It's use case is specifically for large data sets. Which other tools have good Scala support? Is Scala best suited for larger data sets? Or is it also ...
- 1,169
4
votes
1
answer
1k
views
How to set up multi cluster spark without hadoop on Google Compute engine
I'm new to apache spark.
Is it possible to configure multi cluster spark without hadoop?
If so, can you please provide the steps.
I would like to create clusters on Google Compute Engine (1-...
- 101
6
votes
3
answers
1k
views
Summarize and visualize a CSV in Java/Scala?
I would like to summarize (as in R) the contents of a CSV (possibly after loading it, or storing it somewhere, that's not a problem). The summary should contain the quartiles, mean, median, min and ...
- 178