This is a test version of Biostars. For the public version, visit https://www.biostars.org.
Forum: What is your best "If only I'd known about this feature sooner" moment?

Today, after using IGV for 10+ years with Illumina paired-end data, I discovered that you can easily highlight a pair of reads by Ctrl-clicking on one of them. Until now, I had been using the “Go to mate” option in the context menu and frustratingly jumping back and forth for ages.

I suspect that no one in my company of more than 30 people who use IGV daily is actually using this feature the right way.

Tell me about your moments where you discovered a helpful (and maybe poorly documented) feature in bioinformatics software far too late, or realized you’d been doing something the hard way for far too long.

Maybe we can all learn something obvious today ;)

software igv

10 answers

It is possible to combine VS Code and AI plugins to run and "vibe code" local notebooks on HPC, leveraging the ability to process sensitive data with HPC power.

  • Run a jupyter instance on your HPC
  • Create a python/R/Bash... kernel on the HPC
  • Tunnel that jupyter instance by ssh on your local machine
  • Open VS code
  • Inside VS Code, select a notebook
  • Pick the jupyter instance URL as a distant server
  • Select the kernel from your HPC
  • Run your coding cells

The huge advantage for me is that you can use the help of AI on your notebooks locally (like Claude or Codex as VS Code plugins) without giving a direct access to the raw data (as long as you do not output sensitive data in your cells output) but still computing on the HPC.

The cells are processed in the HPC but the interpretation of the cells as well as the rest of the notebook (markdown etc...) are interpreted by your local python system. The AI can read the path of a specific file as a chain of character but will not be able to open it.

The AI has no ability to screw up and look into your raw data on the HPC, as long as your don't specifically allow the AI to tunnel itself into the HPC. Technically I guess it is still possible but less likely.

Edit : In order for this to work, your HPC most allow port fowarding

Not bioinformatics:

unzip -l openoffice_document.odp

cat can be used to concatenate gzip files

cat fastq1.fq.gz  fastq2.fq.gz  > out.gz

paste for linerarizing:

seq 1 20 | paste -sd,

always use input: path in nextflow if you later want to use a container

always use a tuple val(meta), path(file) for your input/output in nextflow to associate metadata with your files

multiMap is the operator of choice in nextflow if your input is a tuple val(meta),path(f1),path(f2) but the process wants

input:
   tuple val(meta1),path(f1)
   tuple val(meta2),path(f2)

'@' is a legal letter for the quality in the fastqs

a awk command can be an pattern without action : awk '$1=="chr1"' input.bed

always use LC_ALL=C when using sort

quoting here-document

cat << 'EOF' 
hello $world
EOF

use set -euo pipefail

R is a terrible language.

I have a hard time with R, many times I said let me learn R in depth ... then I start having to use constructs and R idioms that are so evidently wrong sp many foot guns - I get deeply disillusioned and quit because of how terrible that language is

This blog captures it all

The more you learn about the R language, the worse it will feel.

I suspect that a non-trivial amount of scientific irreproducibility stems from the R language being so easy to misuse.

I have used R as my main work language for years in bioinformatics with Python as a supplement for less supported tasks and increasingly for single cell. I have also taught R to students and staff at my institutes so I'm used to new people coming to the language and the typical pitfalls they experience. I agree that the syntax and functionality of the language will feel weird to people coming from other languages.

To call it 'bad' is weird. That's like saying a foreign language is bad because you're used to Germanic or Romantic languages. I definitely think it's a bit rich to say that it's easy to misuse. I've done plenty of stupid things in Python or JS. If your main goal is to load some packages/libraries, load in some raw data and run through an analysis script, R is much more straightforward than Python and its available visualisation packages as just better.

Also, at least a few of the issues raised in the blog you linked have either been explicitly addressed in newer versions of R or are things that I learned how to correctly work with early in learning R, so I dunno. It seems much more for the sake of humour than an actual critique of R, and there are plenty of reasons to complain about R.

not specifically bioinformatics but process substitution in bash cmdline!

<( command ) 

(not a recent 'revelation' though :) )

One of my all time favourites

Oddly, I've encountered scenarios where this doesn't work. The <() translates to < /proc/fd* and sometimes I get errors about that when the result saved in a file works fine. Maybe there is a time period for that to become stale?

never encountered it myself tbh, but there might be cases where it can happen it seems ...

perhaps when you try to work 'cross-terminal' or so? or background/foreground tasks?

tail -f <some (log)file>

typically for log file viewing: file is kept open and you see the latest entries being added in real-time

Not so much known about, but just bothered with:

Argparse and getopt for python/shell scripting. Something really nice about writing a proper script with proper options rather than relying on sys and shell variables all the time.

Another kind of niche one is that pandas has a 'read fixed width' file function, which when combined with a template allows you to read the otherwise quite verbose outputs that some tools use. In principle, this makes any regularly formatted file interrogable with code in lieu of machine-friendly formats.

template = u"""
---|------|------------------------|----|-------|-------|------|-----|----|---------|--------------|
 No Hit    Short Desc               Prob E-value P-value  Score    SS Cols Query HMM  Template HMM
"""


def hhparse(hhresult_file, verbose):
    """Convert HHpred's text-based output table in to a pandas dataframe"""
    pattern = StringIO(template).readlines()[1]
    colBreaks = [i for i, ch in enumerate(pattern) if ch == "|"]
    widths = [j - i for i, j in zip(([0] + colBreaks)[:-1], colBreaks)]

    hhtable = pd.read_fwf(hhresult_file, skiprows=8, nrows=10, header=0, widths=widths)
    if verbose is True:
        print(hhtable)

    top_hit = str(hhtable.loc[0, "Hit"])[0:4]
    top_hit_full = hhtable.loc[0, "Hit"]
    top_prob = hhtable.loc[0, "Prob"]
    top_eval = hhtable.loc[0, "E-value"]
    top_pval = hhtable.loc[0, "P-value"]
    top_score = hhtable.loc[0, "Score"]

    if verbose is True:
        print("Your best hit: (PDB ID | Probability | E-Value | P-Value | Score)")
        print("\t".join([top_hit, top_prob, top_eval, top_pval, top_score]))

    return top_hit, top_hit_full, top_prob, top_eval, top_pval, top_score

https://github.com/jrjhealey/bioinfo-tools/blob/master/tabulateHHpred.py#L41C1-L68C74

I'm also a big fan of one-liners for getting little jobs done, but sometimes you need a little more firepower, so being able to call on modules like BioPython in a shell one liner is a nice extra little boost:

For example, to robustly linearise a fasta file, you can write a shell function like so:

pylinearisefa(){
python3 -c 'import sys;from Bio import SeqIO; [print(f">{r.id}\n{r.seq}") for r in SeqIO.parse(sys.argv[1], "fasta")];' $1
}

Coding - In gitlab or github instance, press dot . on your keyboard while in a repository to get a fully featured vscode editor for code editing.

Storage space analysis - dust https://github.com/bootandy/dust is super fast and essential for me.

Thanks for sharing the GitHub one, colindaven that is incredibly convenient and I cannot even quantify how many times I've wished I could just make a quick edit on the spot!

Two amazing commands:

meld file1 file2

Essentially diff with a gui

Need to compare latex files for a submission?

latexdiff old.tex new.tex > diff.tex

You mean two amazing binaries - these don't ship as standard with the OS.

I wish I knew about pixi a little sooner. It is a replacement for conda from the people that built conda

https://pixi.prefix.dev/latest/

See also:

A massively underrated tool imo: tableview. With less you get something like:

FIRST AUTHOR    DATE    JOURNAL LINK
Lane JM 2019-02-25      Nat Genet       www.ncbi.nlm.nih.gov/pubmed/30804566
Lane JM 2019-02-25      Nat Genet       www.ncbi.nlm.nih.gov/pubmed/30804566
Nishiyama T     2019-02-22      Sleep   www.ncbi.nlm.nih.gov/pubmed/30810208
Gunjaca I       2019-03-01      J Hum Genet     www.ncbi.nlm.nih.gov/pubmed/30824882
Gunjaca I       2019-03-01      J Hum Genet     www.ncbi.nlm.nih.gov/pubmed/30824882
Gunjaca I       2019-03-01      J Hum Genet     www.ncbi.nlm.nih.gov/pubmed/30824882
Brcic L 2020-02-04      Sci Rep www.ncbi.nlm.nih.gov/pubmed/32019955
Yu D    2019-03-01      Am J Psychiatry www.ncbi.nlm.nih.gov/pubmed/30818990
Bye A   2020-02-05      Prog Cardiovasc Dis     www.ncbi.nlm.nih.gov/pubmed/32035127
Theriault S     2019-10-15      Circ Genom Precis Med   www.ncbi.nlm.nih.gov/pubmed/32141789
Boua PR 2020-02-07      Front Genet     www.ncbi.nlm.nih.gov/pubmed/32117412

with tableview:

FIRST AUTHOR      | DATE       | JOURNAL                          | LINK
Lane JM           | 2019-02-25 | Nat Genet                        | www.ncbi.nlm.nih.gov/pubmed/30804566
Lane JM           | 2019-02-25 | Nat Genet                        | www.ncbi.nlm.nih.gov/pubmed/30804566
Nishiyama T       | 2019-02-22 | Sleep                            | www.ncbi.nlm.nih.gov/pubmed/30810208
Gunjaca I         | 2019-03-01 | J Hum Genet                      | www.ncbi.nlm.nih.gov/pubmed/30824882
Gunjaca I         | 2019-03-01 | J Hum Genet                      | www.ncbi.nlm.nih.gov/pubmed/30824882
Gunjaca I         | 2019-03-01 | J Hum Genet                      | www.ncbi.nlm.nih.gov/pubmed/30824882
Brcic L           | 2020-02-04 | Sci Rep                          | www.ncbi.nlm.nih.gov/pubmed/32019955
Yu D              | 2019-03-01 | Am J Psychiatry                  | www.ncbi.nlm.nih.gov/pubmed/30818990
Bye A             | 2020-02-05 | Prog Cardiovasc Dis              | www.ncbi.nlm.nih.gov/pubmed/32035127
Theriault S       | 2019-10-15 | Circ Genom Precis Med            | www.ncbi.nlm.nih.gov/pubmed/32141789
Boua PR           | 2020-02-07 | Front Genet                      | www.ncbi.nlm.nih.gov/pubmed/32117412

You can also use column:

cat file.tsv | column -st "\t" | less -S

Switch "\t" to , for csv files.

Yes, that's what I've been doing before discovering tableview, but it was quite contrived since I'm lessing tabular files all the time. In fact, I'm surprised there isn't anything more popular... (any suggestion?)

I don't know if this counts as something I wish I'd known sooner, but it definitely cost me a ridiculous amount of time to track down the resolution...

While building a Jupyter notebook for a project, I ran into an IPython display quirk that shows up across Jupyter Notebook, JupyterLab, VSCode, GitHub, and PyCharm. If a cell outputs HTML first (e.g., display(HTML(...)), tabulate, pandas HTML tables) and then ends with a plain‑text object (a list, string, NumPy repr, etc.), the final line gets rendered vertically, as if it were a column vector.

Example:

Expected = ['I','E','I','I','I','E','E']

Actual = 
'I'
'E'
'I'
'I'
'I'
'E'
'E'

The code call was right- the issue was with IPython's MIME‑type priority. Once HTML is emitted, the frontend enters “rich display mode,” and any subsequent plain‑text output inherits the HTML layout unless something forces a flush.

The fix was frustratingly simple: explicitly print the final value. And it must be print(), not display(), because display still emits a rich display MIME bundle, even for a plain string. This forces IPython to close the HTML block and render the text horizontally, as intended.

It's a tiny thing, but I wish I'd known it before losing an afternoon to debugging something that wasn't even broken. Jupyter notebooks are explicitly designed so that simply evaluating an object will auto-display it properly... except in this case. I hope sharing this experience saves someone else the time and frustration I endured.

Because of the complexities it allows, there are a few edge cases. This is certainly an interesting one that I don't recall coming across. I think putting example code that generates your 'Actual' and then the fix may help increase the usefulness here.

Log in to answer this question.