Questions tagged [scala]

Scala is a high level language that combines functional and object oriented programming with high performance runtimes. Scala programming language is build to implement scale able solutions to crunch big data / data science in order to produce actionable insights is a great languages for large-scale projects. Designed to express common programming patterns in a concise, elegant, and type-safe way, it fuses both imperative and functional programming styles.

Filter by
Sorted by
Tagged with
1 vote
1 answer
38 views

Outlier Elimination in Spark With InterQuartileRange Results in Error

I have the following function that is supposed to calculate the outlier for a given dataset. ...
user avatar
  • 137
0 votes
0 answers
31 views

Spark Dataframe APIs vs Spark SQL

I have a relatively complex query which runs against a database and contains multiple join statements, lead/lag functions, subquery, etc. These tables are available as individual files in my object ...
user avatar
0 votes
0 answers
29 views

Find a column by name in a row in scala spark

I have a Seq[Row].Each row is an Array of Struct.Struct has four fields: a,b,c and d all of which are String.The data in a particular row is something like this: [{"a":"ahahk",&...
user avatar
0 votes
0 answers
39 views

Issue while reading a csv file through Spark

I am trying to read a csv file through Spark. However one of the columns has the data in the below format and because of comma it is being split into multiple columns. The input csv file is a comma ...
user avatar
1 vote
2 answers
61 views

unsupervised anomaly detection for univariate fast frequency time series data?

I have a univariate time series (there is a value for each time sampling) (sampling time: 66.66 micro second, number of samples/sampling time=151) coming from a scala customer This time series ...
user avatar
  • 1,526
0 votes
1 answer
22 views

determining size of batch, time of sending and memory in to send from scala to ML section

I have a time series (sampling time: 66.66 micro second, number of samples/sampling time=151), I would like to determine some anomalies in them, the inputs are made by scala customer message bus. ...
user avatar
  • 1,526
1 vote
0 answers
100 views

how to calculate the cosine similarity between two files?

I am using spark and scala to implement an issue. files contain phrases or sentences. I want to use domain based method to calculate the cosine similarity between tags.I convert two files into a ...
user avatar
  • 11
4 votes
1 answer
672 views

Has anyone succeeded in finding a good Scala/Spark kernel for Jupyter?

The ones I've tried so far Almond: Works very well for just Scala, but you have to import dependencies, and it gets tedious after a while. And unfortunately can't run when using Spark with YARN ...
user avatar
1 vote
1 answer
333 views

Can I use model which is trained using Keras to Scala?

I train the NLP classification model using Keras by Python. But to deploy the trained model to a platform I need to use Scala. Can I use Scala with Keras model what I trained using Python? Thanks
user avatar
  • 11
3 votes
3 answers
985 views

How to install Polynote on Windows?

I've been searching around the Internet for a while but I have not been able to find detailed instructions on how to install Polynote (the polyglot notebook with first-class Scala support) for Windows ...
user avatar
  • 2,301
-1 votes
1 answer
78 views

Which language to learn for Machine Learning? [closed]

I am currently working in BigData with Spark-Scala framework. I want to learn Machine learning from scratch. Which language would be better to learn for machine learning, Scala or Python?
user avatar
  • 1
1 vote
0 answers
29 views

Is there a way to use a pom.xml file to update spark configuration?

I am trying to update my spark configuration to solve some dependency problems. This pom.xml file seems to be useful for this purpose. I am using a spark docker image. ...
user avatar
  • 415
1 vote
0 answers
208 views

XGBoost not learning

I have developed a train set for XGBoost to apply a learning to rank function on top of with the following parameters: ...
user avatar
  • 235
1 vote
0 answers
444 views

Spark Scala concatenate 2 different data frames

I have 2 different Spark data frames and I want to concatenate them together by columns with no join operations. How can I do it using Scala?
user avatar
  • 531
0 votes
2 answers
505 views

Apache Spark alternatives for local compute

I am creating a relatively large hobby project in Scala that needs a few ml algorithms for text classification into topics. My dataset is not huge, it is < 500,000 items with dimensionality 5 (2 ...
user avatar
1 vote
1 answer
320 views

Standard Scaler drops accuracy significantly in Scala Spark

I am working on Scala with Spark for a prediction model. I tried both Normalization and Standard Scaling and both of them drops my accuracy significantly. Without the accuracy is ~90% (on training ...
user avatar
  • 3,700
2 votes
1 answer
191 views

XGBoost most important features appear in multiple trees multiple times

I am fitting xgboost model (scala-spark) to my dataset of transactions. I have about 2 millions of transactions in my training set which is highly unbalanced with a ratio of positive/negative<0.001 ...
user avatar
  • 121
0 votes
1 answer
44 views

How does "sameModel" mean in Spark ML Pipeline, (Section: Example: Pipeline), from the docs?

I'm looking at the Spark ML docs in scala for Section: Example Pipeline https://spark.apache.org/docs/latest/ml-pipeline.html#example-pipeline. From the example, the model is fit using a pipeline (...
user avatar
  • 11
1 vote
1 answer
18 views

Scala mllib 2.2.0 k means. Number of runs

As per the documentation here, the runs parameter does nothing. I have looked at the actual Scala code for k means and it appears that the ...
user avatar
  • 111
1 vote
1 answer
4k views

How to apply StandardScaler and OneHotEncoder simultaneously in Spark Machine learning?

I try to create a machine learning model, linear regression, to predict a price of a diamonds. All examples that I found online do not have a step with scaling of data, using MinMaxScaler or ...
user avatar
2 votes
1 answer
368 views

Best practice for developing using Spark [closed]

I am looking for any tips and best practice on how to develop applications using Spark. I currently have access to a cluster, with data as well as a version of Spark 2.1.0 on an edge node and IntelliJ ...
user avatar
  • 478
0 votes
1 answer
196 views

How to Access a Text File from IBM DSX Scala-Spark Notebook [closed]

I have recently faced problem in accessing text files that I have uploaded from my local computer to the IBM DSX Cloud Object Storage so that I can use it in my Spark-Scala notebooks. It appeared to ...
user avatar
1 vote
1 answer
105 views

Multidimensional regression in Scala

I have a queuing model written in Scala where different categories of people end up a different queues. We have a dataset providing a map of features to the numbers of people ending up at each queue, ...
user avatar
5 votes
1 answer
2k views

Saving Large Spark ML Pipeline to HDFS

I'm having trouble saving a large (relative to spark.rpc.message.maxSize) Spark ML pipeline to HDFS. Specifically, when I try to save the model to HDFS, it gives me an error related to spark's maximum ...
user avatar
2 votes
0 answers
78 views

Why does my master node get heap memory full for inbuilt SVD API in Apache Spark during calculation of inverse of a square matrix?

I am using a three node system: Master node: $64$ GB RAM 2 slave node: $32$ GB RAM I am calculating the inverse of a square matrix using inbuilt Apache Spark function SVD. When I called SVD function ...
user avatar
1 vote
0 answers
89 views

How to adapt the LBFGS algorithm to accept a different data input parameter in Spark?

I´m trying to use the LBFGS algorithm to minimize the parameters of a model used for Customer Life Time Value analysis (https://github.com/CamDavidsonPilon/lifetimes). The lifetimes lib in python uses ...
user avatar
-1 votes
1 answer
3k views

Spark: calculating percentage from a non-Int field

I have a spreadsheet of banking information and one of the goals is to find out the failure rate of an advertising campaign. I think I need to get a count of the total number of entries vs. the total ...
user avatar
0 votes
2 answers
2k views

Scala RDD operation

I am new in scala. I have a csv file stored in hdfs. I am reading that file in scala using ...
user avatar
1 vote
1 answer
457 views

Issue with Spark SVD

I have the following dataset with the dimensions: Rows: 41174 Columns: 439316 The matrix is very sparse and on this, I want to perform Dimensionality Reduction. I am using Spark's computeSVD ...
user avatar
5 votes
2 answers
5k views

What is the best deep learning library for scala? [closed]

Does any one has a recommendation for what libraries to use for deep learning?
user avatar
  • 684
3 votes
2 answers
539 views

Hashing trick with random forest in scala

I am trying to perform a hashing trick and then a random forest with scala. I have the following code: ...
user avatar
  • 101
0 votes
1 answer
7k views

Join two dataframes - Spark Mllib

I've two dataframes. The first have the some details from all the students, and the second have only the students that haved positive grade. How can I return only the details of the student that have ...
user avatar
  • 173
3 votes
3 answers
2k views

Task not serializable Error

...
user avatar
2 votes
3 answers
11k views

value saveAsTextFile is not a member of org.apache.spark.sql.DataFrame

How Can I save a DataFrame as a CSV in HDFS? I'm trying with this: df.groupBy("Product_ID").agg(collect_list($"Stock")).saveAsTextFile("PATH/results.csv")
user avatar
0 votes
1 answer
2k views

Apply GroupByKey to identify the Frequently Products Purchase Together

I've a data set with this fields: Transaction_ID Customer_ID Department Product_ID And I'm trying to obtain a tuple with the products associated with each ...
user avatar
  • 173
1 vote
2 answers
2k views

Market Basket Analysis - Data Modelling

Imagine that I've the following dataset: ...
user avatar
  • 173
0 votes
1 answer
2k views

Items in a transaction must be unique but got WrappedArray

I'm trying to do a algorithm in Spark Mllib. I'm trying to do a Market Basket Analysis. And I've as a data set this: ...
user avatar
  • 173
0 votes
1 answer
1k views

How to down-weight non-words in text classification?

Background: Documents coming in as well as training set have gone through Apache Tika with Tesseract for inline images. This works well, except when it doesn't. Many docs are old, scanned images ...
user avatar
4 votes
3 answers
4k views

Scala vs Java if you're NOT going to use Spark?

I'm facing some indecision when choosing how to allocate my scarce learning time for the next few months between Scala and Java. I would like help objectively understanding the practical tradeoffs. ...
user avatar
  • 1,869
2 votes
2 answers
466 views

Spark Scala alternative Machine Learning Library?

I’m using Spark Scala for multiclass classification, and features are continuous. MLlib seems to be limited to Decision Tree and Random Forest for this type of classification – for Naïve Bayes, ...
user avatar
  • 133
5 votes
2 answers
3k views

Distributed k-means in Spark

I want to implement K-means algorithm in Spark. I am looking for a starting point and I found Berkeley's naive implementation. However, is that distributed? I mean I see no mapreduce operations. Or ...
user avatar
  • 291
3 votes
1 answer
2k views

Sampling with replacement, specify the probabilities

I am trying to do sampling with replacement in Scala/Spark, defining the probabilities for each class. This is how I would do it in R. ...
user avatar
  • 31
4 votes
1 answer
2k views

Plotting libraries for Scala on Zeppelin

My main question is it looks like Zeppelin limit the display of the results to on 1000, I know that I can change this number but when I change it Zeppelin become slow. And it looks like the default ...
user avatar
  • 564
14 votes
2 answers
99k views

How to calculate the mean of a dataframe column and find the top 10%

I am very new to Scala and Spark, and am working on some self-made exercises using baseball statistics. I am using a case class create a RDD and assign a schema to the data, and am then turning it ...
user avatar
4 votes
2 answers
4k views

How Mllib in Spark select variables in logistic regression

I have a question about MLlib in Spark.(with Scala) I'm trying to understand how LogisticRegressionWithLBFGS and LogisticRegressionWithSGD work. I usually use SAS or R to do logistic regressions but ...
user avatar
  • 113
15 votes
4 answers
577 views

Data Science Tools Using Scala

I know that Spark is fully integrated with Scala. It's use case is specifically for large data sets. Which other tools have good Scala support? Is Scala best suited for larger data sets? Or is it also ...
user avatar
4 votes
1 answer
985 views

How to set up multi cluster spark without hadoop on Google Compute engine

I'm new to apache spark. Is it possible to configure multi cluster spark without hadoop? If so, can you please provide the steps. I would like to create clusters on Google Compute Engine (1-...
user avatar
6 votes
3 answers
1k views

Summarize and visualize a CSV in Java/Scala?

I would like to summarize (as in R) the contents of a CSV (possibly after loading it, or storing it somewhere, that's not a problem). The summary should contain the quartiles, mean, median, min and ...
user avatar
  • 179