This is a test version of Biostars. For the public version, visit https://www.biostars.org.
Using Amazon Web Services?

Hello,

I am trying to use amazon web services for bioinformatics. I created a linux EC2 instance, but I've found that the space is limited to 10gB. I am attempting to take advantage of the EBS to create a larger hard drive space for me to work with. I am using the AWS console to attach the EBS to EC2 instance, but cannot figure out how to use the attached storage. Can anyone help me with the steps to make this all work?

Thanks!

cloud
• 7,598 views • link
updated by Ram Ram • written by Peter • Peter

10GB space sounds woefully inadequate to any modern computer. Are you sure you have it right? Check here.

• 0 views • link
updated by Ram Ram • written by Istvan Albert Istvan Albert

3 answers

Useful EC2 tutorials for bioinformatics:

  1. Getting started guide: http://docs.amazonwebservices.com/AWSEC2/latest/GettingStartedGuide/
  2. Getting started JAVA: http://aws.amazon.com/articles/Amazon-EC2/3586
  3. Getting started Python: http://aws.amazon.com/articles/Amazon-EC2/3998
  4. EBS (this is MySQL example, but it is very useful and solves your current problems): http://aws.amazon.com/articles/Amazon-EC2/1663
  5. Hadoop tutorial: http://aws.amazon.com/articles/Amazon-EC2/873

To check available space try this:

[root@domU-12-31-39-00-B2-01 ~]# df -k
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/sda1             10321208   1636668   8160252  17% /
/dev/sda2            153899044    192072 145889348   1% /mnt

10Gb is the size of root disk, "other" disk space is in the /mnt directory.

For data analysis I usually use instance disk space ~1Tb. Note that all data will be deleted when you delete instance. And I save all important data to EBS (permanent storage). If you use US zone then network connection with NCBI is fantastic so expensive data keeping in permanent EBS is not necessary. Alternatively you can use public bioinformatics EBS, see full list here http://aws.amazon.com/datasets?_encoding=UTF8&jiveRedirect=1

Available biological public EBS:

  1. 1000 genomes
  2. Human Liver Cohort (Sage Bionetworks)
  3. C57BL/6J by C3H/HeJ Mouse Cross (Sage Bionetworks)
  4. Ensembl Annotated Human Genome Data - for FASTA
  5. Ensembl Annotated Human Genome Data - for MySQL
  6. Illumina - Jay Flatley (CEO of Illumina) Human Genome Data Set
  7. GenBank
  8. YRI Trio Dataset
  9. Ensembl - FASTA Database Files
  10. Unigene
  11. and more
• 0 views • link
updated by Ram Ram • written by Alex Alex

Additionally, if you boot of an EBS-backed image, you can have up boot volumes up to 10 TB

• 0 views • link
written by Mndoci • Mndoci

Just one suggestion: df -h would be more readable :)

• 0 views • link
updated by Ram Ram • written by Michael Schubert Michael Schubert

Just realized I wrote 10 TB .. make that 1 TB

• 0 views • link
written by Mndoci • Mndoci

Maybe you just have to mount the EBS volume,

Here you can get very basics instructions to mount an attached volume to the instance.

Once you've attached the volume to the instance you just have to mount it with "mount" command

HTH

Marina

• 0 views • link
written by Marina Manrique Marina Manrique

If you use an instance store based volume, rather than a EBS backed one, you will have ~300GB to play with at /mnt. However, it will not be persisted when the instance is deallocated.

Since you have 10GB, you are on EBS. This means you will need to attach a larger EBS at /dev/sdx and mount it as Marina points out.

• 0 views • link
written by Casbon • Casbon

Log in to answer this question.

More posts like this