Questions tagged [aws]
Amazon Web Services (AWS) is a popular cloud computing service offered by Amazon.
66
questions
0
votes
0
answers
5
views
How to handle multiple training jobs in AWS Sagemaker Pipelines?
I am trying to create a pipeline for training an image dataset via Sagemaker Pipelines. Based on the examples I understood that for all distinct stages like data preparation, model training, ...
0
votes
0
answers
33
views
Finding the suitable semantic search engine tool for querying an excel/csv file containing data rich in numbers
I have an excel dataset that has the columns: name of the test, user name, data source, total number of records, number of failed records, number of passed records, date of test execution etc. I want ...
- 275
0
votes
0
answers
68
views
Amazon Kendra's performance on data rich in text vs data rich in numbers
I used Amazon's Kendra to build a semantic search engine. I took 2 datasets for this task - csv & excel. The csv dataset is mostly text. The excel dataset is mostly numbers.
The data to be used is ...
- 275
1
vote
0
answers
48
views
Deploying a model with GPU and pay-per-inference
I may have the wrong stack exchange. If that's the case, could someone point me to a stack that could help with this. Anyways...
My backend employs a sentence transformer model from HuggingFace. Since ...
- 141
0
votes
0
answers
14
views
Hosting Models on a Serverless-like format, AWS specific but not exclusively
Similar question here Deploy ML Model on AWS, but more detail provided below:
If I understand correctly, models such as llama and stable diffusion, are all types of Artificial Neural Networks (ANNs). ...
- 101
0
votes
0
answers
25
views
s3 object as corpus_file for gensim Doc2Vec
Is it possible to use a txt or jsonl file in an s3 bucket as the corpus_file input for a gensim Doc2Vec model? I am looking for something of the form:
...
- 101
0
votes
0
answers
501
views
GPT4All with Langchain generating gibberish in RHEL 8
I was trying to create a pipeline using Langchain and GPT4All (gpt4all-converted.bin). The pipeline ran fine when we tried on a windows system. But now when I am trying to run the same code on a RHEL ...
0
votes
0
answers
13
views
Cloud computing solutions for processing big datasets/ML development?
I apologize in advance if my question is vague - I will edit it/follow up if it requires more context. I would also like to add that my experience is with small-ish datasets that can be run on lower-...
2
votes
2
answers
1k
views
Is it possible to work with AWS Sagemaker Studio Notebooks using a local IDE?
Is it possible to work with AWS Sagemaker Studio Notebooks using an IDE such as VSCode or PyCharm/IntelliJ running on my local machine?
And if yes, how?
Ideally I'd like to want to connect to the ...
- 121
1
vote
1
answer
33
views
What is the best approach to deploy N number of ML models as a scalable service in the Cloud?
I've N (~50) number of sentiment models of different languages, which were fine tuned on HggingFace's transformer models. Each of the models as 2-3 GB in size approx. Now, how can I deploy all these ...
0
votes
0
answers
28
views
How to actually begin using my first EC2 instance on AWS in order to run regressions too big for me to run locally
I am using the RStudio Server Amazon Machine Image (AMI) for a collaborative statistical learning research project intended to produce a paper for publication because its computational requirements ...
- 157
0
votes
1
answer
28
views
Which specific AWS service to use for running Benchmark Regressions on datasets far too large to run locally on my laptop [closed]
I am in the middle of a research project with a collaborator in which he has proposed a novel statistical learning processor for optimal variable selection, and I am running the 3 Benchmark Variable ...
- 157
0
votes
1
answer
460
views
Reading a model stored as binary
I'm using a s3 bucket to store a model I trained in python. Since I'm using an s3 bucket I convert the file to binary first and then store it on the bucket.
...
0
votes
0
answers
92
views
identify data type of column from table using NER models
I have structure data with csv or parquet format, I would like to extract the data type of the column by analyzing the data.
when I looked at the NER from Hugging phase transformers, it actually ...
- 1
0
votes
0
answers
180
views
AWS Stepfunctions using SageMaker ScriptProcessor
Hello Data Science/Engineer folks,
I need a help in orchestrating a ML model using AWS Step Functions. There are some tutorials on using StepFunction with Python SDKs, but all of them are focused ...
- 101
0
votes
0
answers
415
views
Loading a model saved in s3 to SageMaker instance
I trained a ML model in SageMaker using Docker Container. After the training is finished, the model is saved as .pkl file which gets written to S3 bucket as .tar file.
I need to use that model to run ...
- 101
1
vote
0
answers
97
views
How speed up pytorch model loading time
I have deployed the background removal model( Pytorch- pre-trained u2net) in aws using lambda and EFS file system and APIgetway.
I have stored my model in efs and loading to the lambda. the model is ...
- 11
1
vote
1
answer
32
views
Deploy ML Model on AWS
So I have trained a ML Model and I need to use it on AWS to make predictions on newly fed in data on a periodic daily basis, and output this data in the form of an attached email.
However I am fairly ...
- 11
1
vote
1
answer
146
views
Best way to get data into Colab Notebook
I am working concurrently with multiple very large datasets (10s-100s of GBs)). I signed up for Colab Pro+ thinking it is the best option. However, I face a significant bottleneck in getting data into ...
- 293
0
votes
1
answer
511
views
Pandas On AWS Sagemaker
Hello Guys i have a question
We want to start working with AWS Sagemaker.
I understand that i can open Jupiter notebook and work
like it was in my computer.
but i know pandas working on single ...
- 35
0
votes
1
answer
310
views
How Can I Process SageMaker Ground Truth NER JSON Output into DataFrame?
So, I've recently created a job using AWS SageMaker Ground Truth for NER purposes, and have received an output in the form a manifest file. I'm now trying to process the manifest file into a dataframe,...
1
vote
0
answers
105
views
Docker File AWS [closed]
I am trying to implement a docker file for Amazon Sagemaker Container,in initial step i am following this link
https://towardsdatascience.com/brewing-up-custom-ml-models-on-aws-sagemaker-e09b64627722
...
- 31
1
vote
2
answers
98
views
When to tune hyperparameters in deep learning
I am currently playing around with different CNN and LSTM model architectures for my multivariate time series classification problem.
I can achieve validation accuracy of better than 50%. I would like ...
- 19
0
votes
2
answers
213
views
OOM memory after kernel restart, was working before
Ran my CNN on a SageMaker notebook and it started training, but I had to restart the kernel due to AWS disconnecting. However when I tried to rerun my code, I received an OOM error, and it never ...
- 451
2
votes
1
answer
1k
views
How to fix Sagemaker's "No finished training job found associated with this estimator" error?
I ran a complete AWS SageMaker Autopilot experiment. I now want to generate batch forecasts using this model but I get the error: "No finished training job found associated with this estimator. ...
- 21
1
vote
1
answer
147
views
What is the benefit of training an ML model with an AWS SageMaker Estimator?
It looks like there are different routes to deploying an ML model on SageMaker. You can:
pre-train a model, create a deployment archive, then deploy
create an estimator, train the model on SageMaker ...
- 223
1
vote
1
answer
138
views
Alternative to EC2 for running ML batch training jobs on AWS
We are building an ML pipeline on AWS, which will obviously require some heavy-compute components including preprocessing and batch training.
Most the the pipeline is on Lambda, but Lambda is known to ...
- 780
2
votes
1
answer
2k
views
How to include lifecycle configuration in Sagemaker studio's user notebook
I want to use lifecycle configuration in Sagemaker studio so that on start of user's notebook it runs the given lifecycle configuration.
My lifecycle configuration will have shell script which will ...
1
vote
0
answers
44
views
not getting point predictions from sagemaker clarify [closed]
I wanted to create shap values for my predictions in sagemaker. I found out that I can use "clarify" functionality in sagemaker to get shap values. However, I want to get point predictions ...
- 71
-2
votes
1
answer
3k
views
ImportError: Pandas requires version '0.3.0' or newer of 's3fs' [closed]
I'm trying to read files from S3, using boto3, pandas, anaconda, but I have the following error:
...
1
vote
2
answers
4k
views
Deploying multiple pre-trained model (tar.gz files) on Sagemaker in a single endpoint
We have followed the following steps:
Trained 5 TensorFlow models in local machine using 5 different training sets.
Saved those in .h5 format.
Converted those into tar.gz (Model1.tar.gz,...Model5.tar....
- 13
1
vote
1
answer
659
views
AWS SageMaker Model as endpoint size limit
Is there a size limit imposed on models deployed on AWS SageMaker as endpoints? I first tried to deploy a simple TensorFlow/Keras Iris classification model by converting to protobuf, tarring the model,...
- 13
1
vote
0
answers
141
views
Sagemaker - XGBOOST rank:ndcg
Does anyone know if the rank:ndcg is available on AWS Sagemaker? I am currently trying to run a model, but it seems like it's not implemented. Am I using an older xgboost version? Kinda new to AWS ...
- 23
1
vote
0
answers
24
views
Real time visualization in AWS
I want to improve my current application. I am using redis using ElastiCache in AWS in order to store some user data from my website.
This solution is not scalable and I want to scale it using Amazon ...
- 151
3
votes
2
answers
7k
views
upload model to S3
I'm using AWS Sage Maker to build my model. I want to store the model in S3 for later use. How do you save your model in S3 with Amazon Sage Maker? I know this seems trivial but I didn't understand ...
- 53
1
vote
0
answers
25
views
Processing XML via StreamXmlRecordReader and Pig in parallel
I'm trying to process a single large XML file with many records and then parse each record in parallel using AWS EMR.
...
- 111
1
vote
0
answers
41
views
Scalable test-to-control matching method needed for Amazon cloud environment
I'm working on a project to port some of my employer's processes from our local Unix servers to the Amazon cloud. One process matches records from a test group to records from a control group using ...
- 111
1
vote
0
answers
19
views
Moving paper documents to cloud (AWS) - [closed]
I'm trying to solve a simple problem and seeking advise/suggestions - I have lot of paper documents in my drawers which I want to store on cloud (S3) as PDFs.
My idea blueprint is -
1) scanning all ...
- 451
1
vote
1
answer
125
views
Using Amazon Personalize to build a Recommendation System
I would like to build a recommendation system based only in the items metadata.
I have an input vector with some desirable topics that the user want to read about, for example: (self-help, yoga, ...
- 161
1
vote
0
answers
34
views
Integrate SageMaker notebook instances into development flow
I often get stuck by the calculation time when I work on a model locally. Even sampling quite heavily sometimes still let the running times very long. Ideally, these times must be of the order of the ...
- 11
1
vote
0
answers
55
views
Word embedding microservices in the cloud
I would like to build nlp classifiers for various tasks such as sentiment analysis, topic modeling, name entity recognition etc. . I realized that most of them only involve simple logistic regression ...
- 55
3
votes
1
answer
58
views
AWS : Workflow for deep learning
I am using my company computer since I don't have another one or linux. Therefore, I am starting to use cloud resources to perform some tasks.
I have a very simple question: Since most cloud ...
- 143
7
votes
1
answer
5k
views
Does AWS Sagemaker continue running when I go offline?
I'm pulling data from web services in an AWS Sagemaker Notebook.
I'd like to be able to close my computer for an hour, reconnect my computer and see that the data collection has continued to run.
...
- 271
0
votes
1
answer
60
views
Intermittent type error when running CNN inside Docker container
I've been working on a simple computer vision API with a few endpoints for grabbing useful information from ebay images. The API lives in a docker container that looks for the h5 files, downloads ...
- 146
0
votes
0
answers
139
views
Where is $SPARK_HOME on an EC2 instance?
I am working on an EC2 instance with anaconda and pyspark pre-installed. My $SPARK_HOME variable is not set. Any ideas on how to locate my spark home?
3
votes
2
answers
4k
views
sklearn and pandas in AWS Lambda
I made a front end where I would like to make REST calls to an AWS Lambda interfaced with AWS API Gateway.
I dumped my model as a pickle file (and so my encoders) which I initially trained locally. ...
- 531
1
vote
1
answer
2k
views
Difference between AWS amazonei_tensorflow and tensorflow conda environments
In AWS SageMaker there are several Conda environments available based on different ML frameworks:
conda info -e
I would like to know the difference between a <...
- 177
2
votes
1
answer
406
views
Should I remove the trend from timeseries when using DeepAR
I saw that for some other algorithms for timeseries data it is advised to remove trend and seasonality before doing the prediction (ex: ARIMA and LSTM)
I figured out from the paper that SageMaker's ...
- 246
4
votes
2
answers
4k
views
Keras neural network entirely different results on two different hardwares (one is AWS)
I have a Keras model. which is defined as follows:
...
- 599
2
votes
2
answers
2k
views
AWS EC2 Segmentation Fault (core dumped) issue with fast-style-transfer
I'm trying to utilize AWS EC2 p2.xlarge instance to convert images using style-transfer code given in this git repo:https://github.com/lengstrom/fast-style-transfer.git, yet when the input file ...
- 181