Questions tagged [aws]

Amazon Web Services (AWS) is a popular cloud computing service offered by Amazon.

Filter by
Sorted by
Tagged with
0
votes
0answers
17 views

Pandas On AWS Sagemaker

Hello Guys i have a question We want to start working with AWS Sagemaker. I understand that i can open Jupiter notebook and work like it was in my computer. but i know pandas working on single ...
0
votes
0answers
8 views

Amazon Personalize Recommendations on Subset of Items

I am playing with AP to build a recommendation system and have a question that I am unable to find answers anywhere. My interactions dataset and items dataset contain several items. But I want the ...
0
votes
1answer
16 views

How Can I Process SageMaker Ground Truth NER JSON Output into DataFrame?

So, I've recently created a job using AWS SageMaker Ground Truth for NER purposes, and have received an output in the form a manifest file. I'm now trying to process the manifest file into a dataframe,...
1
vote
0answers
11 views

Docker File AWS [closed]

I am trying to implement a docker file for Amazon Sagemaker Container,in initial step i am following this link https://towardsdatascience.com/brewing-up-custom-ml-models-on-aws-sagemaker-e09b64627722 ...
0
votes
0answers
19 views

What are the value propositions of Databricks compared with GCP, AWS or Azure?

Especially when considering GCP, the analytics offer from Google is quite interesting. Why would you go with Databricks? GCP has also great integration between tools as well as great support for ML/AI,...
0
votes
0answers
22 views

COVID-19 Rapid Test Result Image Detection

I fine-tuned an Inception V3 model provided in AWS SageMaker to detect COVID-19 Rapid Test Results (see the image below for an example). I provided about 20 pictures of negative and about 20 pictures ...
1
vote
1answer
41 views

When to tune hyperparameters in deep learning

I am currently playing around with different CNN and LSTM model architectures for my multivariate time series classification problem. I can achieve validation accuracy of better than 50 %. I would ...
0
votes
1answer
87 views

OOM memory after kernel restart, was working before

Ran my CNN on a SageMaker notebook and it started training, but I had to restart the kernel due to AWS disconnecting. However when I tried to rerun my code, I received an OOM error, and it never ...
1
vote
1answer
339 views

How to fix Sagemaker's "No finished training job found associated with this estimator" error?

I ran a complete AWS SageMaker Autopilot experiment. I now want to generate batch forecasts using this model but I get the error: "No finished training job found associated with this estimator. ...
0
votes
0answers
86 views

Why does my first Amazon sagemaker autogluon training job fail and keep failing?

I followed the instructions from this article about creating a code-free machine learning pipeline. I already had a working pipeline offline using the same data in TPOT (autoML). I uploaded my data to ...
0
votes
1answer
43 views

What is the benefit of training an ML model with an AWS SageMaker Estimator?

It looks like there are different routes to deploying an ML model on SageMaker. You can: pre-train a model, create a deployment archive, then deploy create an estimator, train the model on SageMaker ...
1
vote
1answer
59 views

Alternative to EC2 for running ML batch training jobs on AWS

We are building an ML pipeline on AWS, which will obviously require some heavy-compute components including preprocessing and batch training. Most the the pipeline is on Lambda, but Lambda is known to ...
2
votes
1answer
850 views

How to include lifecycle configuration in Sagemaker studio's user notebook

I want to use lifecycle configuration in Sagemaker studio so that on start of user's notebook it runs the given lifecycle configuration. My lifecycle configuration will have shell script which will ...
1
vote
0answers
21 views

not getting point predictions from sagemaker clarify [closed]

I wanted to create shap values for my predictions in sagemaker. I found out that I can use "clarify" functionality in sagemaker to get shap values. However, I want to get point predictions ...
-2
votes
1answer
804 views

ImportError: Pandas requires version '0.3.0' or newer of 's3fs' [closed]

I'm trying to read files from S3, using boto3, pandas, anaconda, but I have the following error: ...
1
vote
1answer
2k views

Deploying multiple pre-trained model (tar.gz files) on Sagemaker in a single endpoint

We have followed the following steps: Trained 5 TensorFlow models in local machine using 5 different training sets. Saved those in .h5 format. Converted those into tar.gz (Model1.tar.gz,...Model5.tar....
1
vote
1answer
336 views

AWS SageMaker Model as endpoint size limit

Is there a size limit imposed on models deployed on AWS SageMaker as endpoints? I first tried to deploy a simple TensorFlow/Keras Iris classification model by converting to protobuf, tarring the model,...
1
vote
0answers
74 views

Sagemaker - XGBOOST rank:ndcg

Does anyone know if the rank:ndcg is available on AWS Sagemaker? I am currently trying to run a model, but it seems like it's not implemented. Am I using an older xgboost version? Kinda new to AWS ...
1
vote
0answers
11 views

Real time visualization in AWS

I want to improve my current application. I am using redis using ElastiCache in AWS in order to store some user data from my website. This solution is not scalable and I want to scale it using Amazon ...
2
votes
2answers
2k views

upload model to S3

I'm using AWS Sage Maker to build my model. I want to store the model in S3 for later use. How do you save your model in S3 with Amazon Sage Maker? I know this seems trivial but I didn't understand ...
1
vote
0answers
14 views

Processing XML via StreamXmlRecordReader and Pig in parallel

I'm trying to process a single large XML file with many records and then parse each record in parallel using AWS EMR. ...
1
vote
0answers
40 views

Scalable test-to-control matching method needed for Amazon cloud environment

I'm working on a project to port some of my employer's processes from our local Unix servers to the Amazon cloud. One process matches records from a test group to records from a control group using ...
1
vote
0answers
19 views

Moving paper documents to cloud (AWS) - [closed]

I'm trying to solve a simple problem and seeking advise/suggestions - I have lot of paper documents in my drawers which I want to store on cloud (S3) as PDFs. My idea blueprint is - 1) scanning all ...
1
vote
1answer
66 views

Using Amazon Personalize to build a Recommendation System

I would like to build a recommendation system based only in the items metadata. I have an input vector with some desirable topics that the user want to read about, for example: (self-help, yoga, ...
1
vote
0answers
33 views

Integrate SageMaker notebook instances into development flow

I often get stuck by the calculation time when I work on a model locally. Even sampling quite heavily sometimes still let the running times very long. Ideally, these times must be of the order of the ...
1
vote
0answers
43 views

Word embedding microservices in the cloud

I would like to build nlp classifiers for various tasks such as sentiment analysis, topic modeling, name entity recognition etc. . I realized that most of them only involve simple logistic regression ...
3
votes
1answer
55 views

AWS : Workflow for deep learning

I am using my company computer since I don't have another one or linux. Therefore, I am starting to use cloud resources to perform some tasks. I have a very simple question: Since most cloud ...
2
votes
1answer
2k views

Does AWS Sagemaker continue running when I go offline?

I'm pulling data from web services in an AWS Sagemaker Notebook. I'd like to be able to close my computer for an hour, reconnect my computer and see that the data collection has continued to run. ...
0
votes
1answer
45 views

Intermittent type error when running CNN inside Docker container

I've been working on a simple computer vision API with a few endpoints for grabbing useful information from ebay images. The API lives in a docker container that looks for the h5 files, downloads ...
0
votes
0answers
85 views

Where is $SPARK_HOME on an EC2 instance?

I am working on an EC2 instance with anaconda and pyspark pre-installed. My $SPARK_HOME variable is not set. Any ideas on how to locate my spark home?
3
votes
2answers
4k views

sklearn and pandas in AWS Lambda

I made a front end where I would like to make REST calls to an AWS Lambda interfaced with AWS API Gateway. I dumped my model as a pickle file (and so my encoders) which I initially trained locally. ...
1
vote
1answer
1k views

Difference between AWS amazonei_tensorflow and tensorflow conda environments

In AWS SageMaker there are several Conda environments available based on different ML frameworks: conda info -e I would like to know the difference between a <...
1
vote
0answers
286 views

Should I remove the trend from timeseries when using DeepAR

I saw that for some other algorithms for timeseries data it is advised to remove trend and seasonality before doing the prediction (ex: ARIMA and LSTM) I figured out from the paper that SageMaker's ...
4
votes
2answers
4k views

Keras neural network entirely different results on two different hardwares (one is AWS)

I have a Keras model. which is defined as follows: ...
2
votes
2answers
2k views

AWS EC2 Segmentation Fault (core dumped) issue with fast-style-transfer

I'm trying to utilize AWS EC2 p2.xlarge instance to convert images using style-transfer code given in this git repo:https://github.com/lengstrom/fast-style-transfer.git, yet when the input file ...
1
vote
1answer
136 views

How can I split an image into rectangles?

I have a labelled form to which people will add their name and a series of numbers. They will then take a picture of the form. Like so: I can get decent results by simply sending this to AWS ...
1
vote
0answers
438 views

Get deepAR fit() metrics to python

I use deepAR RNN on AWS via python. I run the estimator.fit(inputs=data_channels) and want to get insight on the algorithm performance and the fit that the model ...
0
votes
2answers
290 views

deepAR RNN from AWS Sagemaker - should I clean the data first?

I test the Sagemaker AWS solution for RNN: deepAR. Previously I used sklearn for this and obviously I cleaned the data to avoid highly correlated time series (KBest,...
3
votes
2answers
2k views

How do I get the name of Sagemaker Estimator's job

I'm having a stumbling block with SageMaker. How do I know what my job name is? For example: ...
3
votes
1answer
3k views

Why is Amazon P2 instance cost much more than G3?

I'm currently looking for the correct number of 2018 budget for the AWS instances in preparation of more deep neural net training, but now a bit confused because of this chart below: p2.16xlarge ...
0
votes
0answers
78 views

Maximum Dimensionality of AWS Machine Learning

I am toying with AWS Machine Learning and I have a dataset with about 200 records with about 220,000 variables each. Apparently, AWS Machine Learning has a hard limit of 1000 variables. How can I ...
0
votes
2answers
5k views

Does any cloud service provide free GPU access?

Is there any cloud developer platform which provides free access to an NVIDIA GPU instance, maybe only for a limited trial period? I'd like to work on it before committing to a paid option. I have ...
3
votes
1answer
575 views

Using AWS ML to recommend products

I have millions of user ratings on about 2k products. I want to use Machine Learning to analyse these ratings and recommend products to users based on other users ratings of the same and different ...
4
votes
2answers
616 views

AWS machine learning prediction schema problems

I've trained an AWS Machine Learning model with the training data from here : https://www.kaggle.com/c/titanic/data I'm now trying to run a batch prediction with the test data from the same source ...
1
vote
1answer
723 views

How do I access data on my EBS Volume from R-Studio Server on Ubuntu EC2 Instance

I have setup R-Studio Server on an Ubuntu EC2 instance for the first time and successfully started r-studio server in my browser. I also have putty ssh client. How do I set path in r-studio server to ...
13
votes
5answers
21k views

Downloading a large dataset on the web directly into AWS S3

Does anyone know if it's possible to import a large dataset into Amazon S3 from a URL? Basically, I want to avoid downloading a huge file and then reuploading it to S3 through the web portal. I ...
0
votes
1answer
1k views

Can hadoop with Spark be configured with 1GB RAM

I'm trying to set up a cluster (1 namenode, 1 datanode) on AWS. I'm using free one year trial period of AWS, but the challenge is, instance is created with 1GB of RAM. As I'm a student, I cannot ...
6
votes
2answers
1k views

Processing data stored in Redshift

We're currently using Redshift as our data warehouse, which we're very happy with. However, we now have a requirement to do machine learning against the data in our warehouse. Given the volume of data ...
5
votes
3answers
467 views

What technologies are fastest at performing joins on large datasets?

By "large", I mean in the range of 100m to 10b rows. I'm currently using both Hadoop MapReduce and Amazon RedShift. MapReduce has been a little disappointing here. Redshift works very well if the ...
12
votes
3answers
3k views

Does Amazon RedShift replace Hadoop for ~1XTB data?

There is plenty of hype surrounding Hadoop and its eco-system. However, in practice, where many data sets are in the terabyte range, is it not more reasonable to use Amazon RedShift for querying ...